search

b-thorne / PySM_public

PySM: Software for simulating the Galactic microwave sky
24 stars 29 forks source link

Reduce PySM package size #24

Open zonca opened 6 years ago

zonca commented 6 years ago

hi! the current 2.1.0 release tgz file at https://github.com/bthorne93/PySM_public/releases is 380MB. This is quite big and is causing some random unit tests of TOAST to fail because the download from Github takes too long.

I wonder if it would be a good idea to store the maps in a separate Github repository and then use astropy.utils.data functionality to download it only when it is necessary (then cache it).

I recently implemented something similar in healpy to download pixel weights when needed: https://github.com/healpy/healpy/pull/442/files

b-thorne commented 6 years ago

Hi @zonca !

This would be a great feature to add, especially when the products become larger. In the future it is possible that individual files will need to be larger than the 100MB limit, do you have experience on how to handle downloading that either from Github or an alternative?

zonca commented 6 years ago

yes, I think that if soon files could be larger than 100MB better avoid Github. I wonder if we could host them on figshare, so that they would be versioned and easily available separately. I am not sure if that would work with astropy.data.utils, I'll need to test it.

@bthorne93 do you think figshare is an option or you'd prefer some institutional server? If figshare is ok, I can make a test and see if I can use it with astropy.data.utils.

b-thorne commented 6 years ago

I have not used figshare before, but if its hosted content can be made public, that might be easier than an institutional server? Let's see if it can work with astropy then.

On Thu, Jun 28, 2018, 03:27 Andrea Zonca notifications@github.com wrote:

yes, I think that if soon files could be larger than 100MB better avoid Github. I wonder if we could host them on figshare, so that they would be versioned and easily available separately. I am not sure if that would work with astropy.data.utils, I'll need to test it.

@bthorne93 https://github.com/bthorne93 do you think figshare is an option or you'd prefer some institutional server? If figshare is ok, I can make a test and see if I can use it with astropy.data.utils.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bthorne93/PySM_public/issues/24#issuecomment-400989288, or mute the thread https://github.com/notifications/unsubscribe-auth/AQHddIVxexJPWqoIPP6r0w0tlEv8YFj-ks5uBK-dgaJpZM4UrdTq .

zonca commented 6 years ago

it would be useful to provide input templates at all nsides between 512 and 8192, downloaded when requested by the user