search

sstcam / sstcam_nsb

Issue tracking and script storage for SSTCAM investigations of night sky background
2 stars 0 forks source link

Figure out way of distributing Fits files #11

Closed STSpencer closed 3 years ago

STSpencer commented 3 years ago

Given the fits files are large (800mb) and costly to generate, try to figure out a way of sharing these with the rest of SSTCAM for the purposes of reproducability. Github's probably out as the file size limits 100mb. Does anyone know what the file size limit on owncloud is?

RichardWhite109 commented 3 years ago

There's no limit on Nextcloud, but the MPIK cluster would be better if there are many GBs. But - can't these be reduced? For a given field with the camera, this is 2048 numbers? I guess the question is at what level do others want to access the data? For me at least, I'm happy with just having the 'final' photons per camera pixel for a given exposure.

RichardWhite109 commented 3 years ago

For up to a few GB (please no more than ~4 GB) you can put things here:https://pcloud.mpi-hd.mpg.de/index.php/s/wsHcLNDC54EYkRS

STSpencer commented 3 years ago

So yes, for a given field the final output from my part of the code is just an astropy table with 2048 numbers and sky pixel positions, that's what the fluxes.csv files are per investigation in the results section of this github, i.e.:

image

However, the fits files needed for the intermediate step, before we sum the brightness per camera pixel are larger. I didn't write that particular bit of code, it's part of NSB itself, but it includes healpix metadata per pixel etc which runs up the size, plus the pixel maps for that part of the code are necessarily higher density than the camera (see #1). They look like this in ds9:

image

If it's for the purposes of validating my code, I'd rather share the unmodified output of NSB as it's already been through a validation process with H.E.S.S., so we can be at least a bit sure I've not messed it up.

STSpencer commented 3 years ago

For up to a few GB (please no more than ~4 GB) you can put things here:https://pcloud.mpi-hd.mpg.de/index.php/s/wsHcLNDC54EYkRS

Alright sounds good, I'll just upload the current Draco run (https://github.com/sstcam/sstcam_nsb/tree/main/results/draco1) fits file (763mb) to nextcloud so other people can try running the extraction code (fromfits.py) for cross-checks for the time being.

STSpencer commented 3 years ago

Right, I've backed up all the maps files to glamdring, so they'll be safe if the Denys Wilkinson Building collapses/floods. For anything other than that the nextcloud should suffice.