search

MannLabs / alphabase

Infrastructure of AlphaX ecosystem
https://alphabase.readthedocs.io
Apache License 2.0
31 stars 9 forks source link

Mmap io #14

Closed swillems closed 2 years ago

swillems commented 2 years ago

hdf.io datasets now have a .mmap option in addition to their .values option. This only works on contiguous arrays though, therefore the arrays always needs to be created at first usage, greatly expanding the hdf files.

swillems commented 2 years ago

The nbdev keeps messing up my system for alphabase. Feel free to checkout the branch and run it locally on a system that is less buggy than mine and recommit...

swillems commented 2 years ago

Mostly imports indeed. But the gitconfig keeps messing me up for some reason as well. It is often easier to remove it. This does mean I do not automatically clean nbs though, but I can easily enough run this manually if needed (but since I do little to no nb development, this is rarely an issue)

swillems commented 2 years ago

Can you remove certain commits through github, i.e. can I say I just want to merge the io.hdf file and nothing else?

straussmaximilian commented 2 years ago

Ideally one would just not commit the changes. It should be possible to update the branch and only to update the io.hdf file -but would probably not remove the commits and would have to dig a bit into Github. Shall I go ahead an do this?

swillems commented 2 years ago

But not commiting at the very least the .gitconfig change does not allow me to switch branches or push/pull unfortunately (ar at least via atom, some things might work better over terminal, but even this seems wonky sometimes)... So that one is a no go for me. If you have exoeruence with this, feel free to fix it! Otherwise I guess I can manually try to do some changes on my end.

jalew188 commented 2 years ago

It is necessary to merge it now?