Closed STSpencer closed 3 years ago
There's no limit on Nextcloud, but the MPIK cluster would be better if there are many GBs. But - can't these be reduced? For a given field with the camera, this is 2048 numbers? I guess the question is at what level do others want to access the data? For me at least, I'm happy with just having the 'final' photons per camera pixel for a given exposure.
For up to a few GB (please no more than ~4 GB) you can put things here:https://pcloud.mpi-hd.mpg.de/index.php/s/wsHcLNDC54EYkRS
So yes, for a given field the final output from my part of the code is just an astropy table with 2048 numbers and sky pixel positions, that's what the fluxes.csv files are per investigation in the results section of this github, i.e.:
However, the fits files needed for the intermediate step, before we sum the brightness per camera pixel are larger. I didn't write that particular bit of code, it's part of NSB itself, but it includes healpix metadata per pixel etc which runs up the size, plus the pixel maps for that part of the code are necessarily higher density than the camera (see #1). They look like this in ds9:
If it's for the purposes of validating my code, I'd rather share the unmodified output of NSB as it's already been through a validation process with H.E.S.S., so we can be at least a bit sure I've not messed it up.
For up to a few GB (please no more than ~4 GB) you can put things here:https://pcloud.mpi-hd.mpg.de/index.php/s/wsHcLNDC54EYkRS
Alright sounds good, I'll just upload the current Draco run (https://github.com/sstcam/sstcam_nsb/tree/main/results/draco1) fits file (763mb) to nextcloud so other people can try running the extraction code (fromfits.py) for cross-checks for the time being.
Right, I've backed up all the maps files to glamdring, so they'll be safe if the Denys Wilkinson Building collapses/floods. For anything other than that the nextcloud should suffice.
Given the fits files are large (800mb) and costly to generate, try to figure out a way of sharing these with the rest of SSTCAM for the purposes of reproducability. Github's probably out as the file size limits 100mb. Does anyone know what the file size limit on owncloud is?