Update legacy zipshelve

iabraham / pyhcp

Using Human Connectome Data with Python for ML

5 stars 1 forks source link

Update legacy zipshelve #3

Open iabraham opened 5 years ago

iabraham commented 5 years ago

A minimally working (on UNIX) implementation of shelve with zip has been ported over from here. The linked file is written in legacy Python2 format and follows different PEP and coding conventions.

Its been minimally modified to work on Python3 environment of this repository, however:

A stylistic and functional update is highly recommended. For e.g. Utilizing pathlib instead of string operations to parse file names and check if they exist or not.
The context clause usage, (i.e. with something as athing:) seems to work, but needs to be double checked to make sure nothing will break.

iabraham commented 5 years ago

There could also be a python bug. Because I don't seem to be able to open a shelf in readonly mode.

iabraham commented 5 years ago

The bug is resolved in Ubuntu as per here and links therein.

iabraham commented 5 years ago

On MacOS and with then ndbm module there is a known bug on filesize/memory limitation: see here, here and here. This does not seem to be a problem on Windows or Linux.

Possible fixes:

Might have to dump shelve or stick to Linux.
Ask OSX users to install gdbm via brew install gdbm(see here) and use that instead

The second approach will likely be better since the Campus Cluster usually runs Linux.