search

henjo / libpsf

PSF simulation data c++ library
GNU Lesser General Public License v3.0
26 stars 23 forks source link

Invalid chunk 2147483647 #3

Closed taimurrabuske closed 9 years ago

taimurrabuske commented 13 years ago

With files bigger than determined size, the lib returns "Invalid chunk 2147483647" when reading the psf. This number is exactly equal to 2^31−1, so maybe this has to do with using 32bit variables in referencing the file chunk pointer.

henjo commented 13 years ago

Could be, what do mean by bigger than determined size? How big is the PSF file? Anyways all offset references in the PSF files are 32bit numbers and I think there is a maximum file size limit of 2^31 bytes.

Try compile it with debug info and run gdb on it: export CXXFLAGS="-O0 -g -pg" export CFLAGS="-O0 -g -pg" ./configure --prefix=$HOME make clean; make install

gdb python

set args yourpythoncode.py break translate_exception run

And hopefully gdb will break when the exception is thrown. And then to see the backtrace, write

bt

taimurrabuske commented 13 years ago

Hello, thanks for the help: here is the result of gdb backtrace. I will dive into the details as soon as I have some gap in the schedule.

Have a nice week.

Breakpoint 1, translate_exception (e=...) at psfpython.cc:91 91 void translate_exception(IncorrectChunk const& e) { (gdb) bt

0 translate_exception (e=...) at psfpython.cc:91

1 0x00007ffff0bbfad5 in operator() (function_obj_ptr=,

a0=<value optimized out>, a1=<value optimized out>)
at /usr/include/boost/python/detail/translate_exception.hpp:59

2 operator()<bool,

boost::python::detail::translate_exception<IncorrectChunk, void (*)(IncorrectChunk const&)>, boost::_bi::list2<boost::python::detail::exception_handler const&, boost::function0 const&> > ( function_obj_ptr=, a0=, a1=) at /usr/include/boost/bind/bind.hpp:375

3 operator()<boost::python::detail::exception_handler,

boost::function0 > (function_obj_ptr=, a0=, a1=) at /usr/include/boost/bind/bind_template.hpp:102

4 boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool,

boost::python::detail::translate_exception<IncorrectChunk, void ()(IncorrectChunk const&)>, boost::_bi::list3boost::arg<1, boost::arg<2>, boost::_bi::value<void ()(IncorrectChunk const&)> > >, bool, boost::python::detail::exception_handler const&, boost::function0 const&>::invoke ( function_obj_ptr=, a0=, a1=) at /usr/include/boost/function/function_template.hpp:131

5 0x00007ffff0980a94 in boost::function2<bool,

boost::python::detail::exception_handler const&, boost::function0 const&>::operator()(boost::python::detail::exception_handler const&, boost::function0 const&) const () ---Type to continue, or q to quit--- from /usr/lib64/libboost_python.so.5

6 0x00007ffff097ffeb in

boost::python::handle_exception_impl(boost::function0) () from /usr/lib64/libboost_python.so.5

7 0x00007ffff09763cb in ?? () from /usr/lib64/libboost_python.so.5

8 0x000000375b043ae3 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0

9 0x000000375b0dbe90 in PyEval_EvalFrameEx ()

from /usr/lib64/libpython2.6.so.1.0

10 0x000000375b0de561 in PyEval_EvalCodeEx ()

from /usr/lib64/libpython2.6.so.1.0

11 0x000000375b0de672 in PyEval_EvalCode ()

from /usr/lib64/libpython2.6.so.1.0

12 0x000000375b0f964c in ?? () from /usr/lib64/libpython2.6.so.1.0

13 0x000000375b0f9720 in PyRun_FileExFlags ()

from /usr/lib64/libpython2.6.so.1.0

14 0x000000375b0faafc in PyRun_SimpleFileExFlags ()

from /usr/lib64/libpython2.6.so.1.0

15 0x000000375b10717d in Py_Main () from /usr/lib64/libpython2.6.so.1.0

16 0x0000003dd521ec5d in __libc_start_main () from /lib64/libc.so.6

17 0x0000000000400649 in _start ()

On Mon, Apr 4, 2011 at 6:43 AM, henjo reply@reply.github.com wrote:

Could be, what do mean by bigger than determined size? How big is the PSF file? Anyways all offset references in the PSF files are 32bit numbers and I think there is a maximum file size limit of 2^31 bytes.

Try compile it with debug info and run gdb on it: export CXXFLAGS="-O0 -g -pg" export CFLAGS="-O0 -g -pg" ./configure --prefix=$HOME make clean; make install

gdb python

set args yourpythoncode.py break translate_exception run

And hopefully gdb will break when the exception is thrown. And then to see the backtrace, write

bt

Reply to this email directly or view it on GitHub: https://github.com/henjo/libpsf/issues/3#comment_952628

Taimur Gibran Rabuske Kuntz Researcher at Analog and Mixed-Signal Circuits Research Unit INESC-ID - Instituto de Engenharia de Sistemas e Computadores - Investigação e Desenvolvimento

Lisbon - Portugal

GPG keyID: D4D0FEB8 @ pgp.mit.edu

taimurrabuske commented 13 years ago

I couldn't solve the problem, but maybe declaring the offset with uint64_t variables would solve? Maybe use the code snippet below

define ntohll(x) ( ( (uint64_t)(ntohl( (uint32_t)((x << 32) >> 32) )) << 32) | ntohl( ((uint32_t)(x >> 32)) ) )

define GET_INT64(buf) ntohll(((uint64_t )(buf)))

in place of GET_INT32()?

ghost commented 9 years ago

Hi Henjo,

I played around with libpsf, but taimurgibran is right: it breaks for small to medium sized PSF files with the same bad-chunk error. It is not only files greater than 2^31 bytes (which would be pretty big 2+ GB files). It also breaks for PSF files as small as 20MB. I can send a test file if anyone wants to figure out a solution. It looks like 32-bit datatypes are pretty deeply entwined in the code, so doing something simple like changing 32 to 64 in variable declarations doesn't seem to work. I haven't spent a lot of time trying to understand the problem though.

-Paul

henjo commented 9 years ago

Hi Paul,

32-bit integers are handled by the PSFInt32Scalar class which is using the PSFScalarT template. But it's hard to say what the problem is. Please send me a data file that gives the problem. I guess the file would be small enough to send over email if it's compressed.

/Henrik

ghost commented 9 years ago

Hi Henrik,

Thanks for the response! I tried to send the test file by responding to your github email, but that message exceeded their email server's size limit. The file is 22MB, or about 10MB compressed into a .tar.gz. If you can send me your email address somehow, I can send you the test case.

-Paul

henjo commented 9 years ago

Sure, my email is henrik[@]johome[dot]net.

henjo commented 9 years ago

I finally found this problem and it was quite serious. One could not get any signals from windowed data (such as transient analysis) unless the last signal was requested :). Please try the latest version to see if the latest commit solves the problem.

ghost commented 9 years ago

Seems to work now! Thanks for the help.

henjo commented 9 years ago

Great, thanks for helping with tracking down this bug.

On Friday, October 10, 2014 09:35:41 paulswirhun wrote:

Seems to work now! Thanks for the help.

Reply to this email directly or view it on GitHub:

https://github.com/henjo/libpsf/issues/3#issuecomment-5868 1038