bshillingford / python-torchfile

Deserialize Lua torch-serialized objects from Python
BSD 3-Clause "New" or "Revised" License
216 stars 25 forks source link

Windows is not supported? #12

Closed xiaoyongzhu closed 2 years ago

xiaoyongzhu commented 6 years ago

I am using this torch file (https://s3.amazonaws.com/xunhuang-public/adain/vgg_normalised.t7) and is using the following line of code to load it:

t7 = torchfile.load(t7_file, force_8bytes_long=False)

Looks like it works in Linux, but it doesn't work in Windows. Error message is:

    t7 = torchfile.load(t7_file, force_8bytes_long=False)
  File "C:\Program Files\Python36\lib\site-packages\torchfile.py", line 424, in load
    return reader.read_obj()
  File "C:\Program Files\Python36\lib\site-packages\torchfile.py", line 370, in read_obj
    obj._obj = self.read_obj()
  File "C:\Program Files\Python36\lib\site-packages\torchfile.py", line 385, in read_obj
    k = self.read_obj()
  File "C:\Program Files\Python36\lib\site-packages\torchfile.py", line 386, in read_obj
    v = self.read_obj()
  File "C:\Program Files\Python36\lib\site-packages\torchfile.py", line 370, in read_obj
    obj._obj = self.read_obj()
  File "C:\Program Files\Python36\lib\site-packages\torchfile.py", line 387, in read_obj
    obj[k] = v
TypeError: unhashable type: 'list'

Looks like something is wrong here? How should I fix it?

bshillingford commented 6 years ago

Ah, this is likely because a table is used to index another table in the original lua. Unrelated to Windows.

Try turning off the list heuristic...it's another keyword arg. Everything will be a dict, but it should work.

On Nov 30, 2017 12:42 AM, "Xiaoyong Zhu" notifications@github.com wrote:

I am using this torch file (https://s3.amazonaws.com/ xunhuang-public/adain/vgg_normalised.t7) and is using the following line of code to load it:

t7 = torchfile.load(t7_file, force_8bytes_long=False)

Looks like it works in Linux, but it doesn't work in Windows. Error message is:

t7 = torchfile.load(t7_file, force_8bytes_long=False)

File "C:\Program Files\Python36\lib\site-packages\torchfile.py", line 424, in load return reader.read_obj() File "C:\Program Files\Python36\lib\site-packages\torchfile.py", line 370, in read_obj obj._obj = self.read_obj() File "C:\Program Files\Python36\lib\site-packages\torchfile.py", line 385, in read_obj k = self.read_obj() File "C:\Program Files\Python36\lib\site-packages\torchfile.py", line 386, in read_obj v = self.read_obj() File "C:\Program Files\Python36\lib\site-packages\torchfile.py", line 370, in read_obj obj._obj = self.read_obj() File "C:\Program Files\Python36\lib\site-packages\torchfile.py", line 387, in read_obj obj[k] = v TypeError: unhashable type: 'list'

Looks like something is wrong here? How should I fix it?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bshillingford/python-torchfile/issues/12, or mute the thread https://github.com/notifications/unsubscribe-auth/ACOA3RQlBUQScvQq-K3a6M4d289-Uqb7ks5s7fndgaJpZM4Qv2uq .

xiaoyongzhu commented 6 years ago

@bshillingford Thanks for the reply! I have turned off it:

    t7 = torchfile.load(t7_file, force_8bytes_long=False, use_list_heuristic = False)

However there's still some problem:

    obj._obj = self.read_obj()
  File "C:\Program Files\Python36\lib\site-packages\torchfile.py", line 385, in read_obj
    k = self.read_obj()
  File "C:\Program Files\Python36\lib\site-packages\torchfile.py", line 386, in read_obj
    v = self.read_obj()
  File "C:\Program Files\Python36\lib\site-packages\torchfile.py", line 370, in read_obj
    obj._obj = self.read_obj()
  File "C:\Program Files\Python36\lib\site-packages\torchfile.py", line 387, in read_obj
    obj[k] = v
TypeError: unhashable type: 'numpy.ndarray'

which is in the same line.

Looks like it's still not working. BTW - the same piece of code and data can work in Ubuntu, but not in Windows, so i suspect it's something related with the OS.

bshillingford commented 6 years ago

Odd. Sorry, I don't have windows so I can't test.

On Nov 30, 2017 02:20, "Xiaoyong Zhu" notifications@github.com wrote:

@bshillingford https://github.com/bshillingford Thanks for the reply! I have turned off it:

t7 = torchfile.load(t7_file, force_8bytes_long=False, use_list_heuristic = False)

However there's still some problem:

obj._obj = self.read_obj()

File "C:\Program Files\Python36\lib\site-packages\torchfile.py", line 385, in read_obj k = self.read_obj() File "C:\Program Files\Python36\lib\site-packages\torchfile.py", line 386, in read_obj v = self.read_obj() File "C:\Program Files\Python36\lib\site-packages\torchfile.py", line 370, in read_obj obj._obj = self.read_obj() File "C:\Program Files\Python36\lib\site-packages\torchfile.py", line 387, in read_obj obj[k] = v TypeError: unhashable type: 'numpy.ndarray'

which is in the same line.

Looks like it's still not working. BTW - the same piece of code and data can work in Ubuntu, but not in Windows, so i suspect it's something related with the OS.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bshillingford/python-torchfile/issues/12#issuecomment-348061938, or mute the thread https://github.com/notifications/unsubscribe-auth/ACOA3X1SDQ9UzS1x7-UGVYrlnFmNU5zRks5s7hD3gaJpZM4Qv2uq .

xiaoyongzhu commented 6 years ago

I can help to host a windows VM on the cloud for you to test if you want to fix it (sorry I really don't have experience in dealing with Lua/Torch so can't help much here). You can find my email in my github introduction page.

Dunimon commented 5 years ago

I tried to load the same model (https://s3.amazonaws.com/xunhuang-public/adain/vgg_normalised.t7) with the command

 torch_model = torchfile.load(t7_file, force_8bytes_long=True)

and it seems working on windows. Honestly, I don't know what the parameters force_8bytes_long is really related to. I mean, the size of the long type is 8 bytes and (do not hesitate to tell me if I'm wrong) that's the case for any languages (lua, python etc..). There is no full documentation (not yet I think) of torchfile libary but we can try figure out what is going on (https://github.com/bshillingford/python-torchfile).

soumith commented 5 years ago

@Dunimon on Windows the size of long is 4 bytes.

dexter2406 commented 3 years ago

I tried to load the same model (https://s3.amazonaws.com/xunhuang-public/adain/vgg_normalised.t7) with the command

 torch_model = torchfile.load(t7_file, force_8bytes_long=True)

and it seems working on windows. Honestly, I don't know what the parameters force_8bytes_long is really related to. I mean, the size of the long type is 8 bytes and (do not hesitate to tell me if I'm wrong) that's the case for any languages (lua, python etc..). There is no full documentation (not yet I think) of torchfile libary but we can try figure out what is going on (https://github.com/bshillingford/python-torchfile).

This works, but it requires python2.x, because it involves xrange... hmmm