search

ghcollin / tftables

HDF5 interface for Tensorflow.
MIT License
74 stars 10 forks source link

_pickle.PicklingError: Can't pickle #8

Open sjsy opened 6 years ago

sjsy commented 6 years ago

Hi, when I ran the test unit on my system, I got the following errors:

_pickle.PicklingError: Can't pickle <function Streamer.__read_process at 0x000002012024BF28>: attribute lookup Streamer.__read_process on multitables failed

Please help me.

ghcollin commented 6 years ago

What is the full output? (stack trace and any HDF5 errors)

QCmonk commented 6 years ago

I have a similiar problem. I have attached the output of running the testing suite. error_log.txt

ghcollin commented 6 years ago

Thanks for the error log. I think I may have found that the problem lies in multitables. It seems older versions of python don't support pickling static methods of classes.

If you are running a version older than 3.5.2, can you try finding where "multitables.py" is installed on your system and replacing it with the version in this link?

And because I found a regression in tftables when using python3 while debugging this, you may want to update "tftables.py" to the version in the link as well, if the unit tests still fail.

https://gist.github.com/ghcollin/7fbfaa78b540b4ab9320c78134f110f1

QCmonk commented 6 years ago

Thank you for the rapid debug. I have done as you suggested, updating both multitables and tftables using the suggested link. Unfortunately I am still failing the unit tests however (log file attached, I doubt the tensorflow warnings are important but it is true heresy to omit such things from an error log). The testing seems to hang which is perhaps the recursion issue? I'll try running in a >=3.5.2 environment and will update iff it works

error_log_2.txt

ghcollin commented 6 years ago

It seems you're running on Windows. In theory, multitables should run on Windows, but I've never set up a VM to try it out. Windows and Linux/OSX have different threading/processing models, so it's possible there is some deeper incompatibility here.

Could you try downloading and running the unit tests for multitables, just to see if it can isolate the problem? You can find them here: https://github.com/ghcollin/multitables/blob/master/multitables_test.py

Could you also find out what version of python you're running? Just in case I need to start debugging that particular version.

Thanks.

QCmonk commented 6 years ago

Ah of course. While I would ideally change my OS I am sadly constrained to Windows in this instance. I am running python 3.5.4 and the multitables test outputs the attached log file.

multitables_log.txt

The rval assertion on line 215 of multitables.py seems to be the root of the problem.

ghcollin commented 6 years ago

Sorry I haven't had a lot of time to look into this lately. It's a definite possibility that the way windows handles shared memory or condition variables is causing the corruption of the internal queues in multitables, as that assertion should never trigger if the queue is properly synchronised. I should hopefully have more time to look into this in a few weeks.