Fit Files - automated move to Amazon S3 bucket

[gaitskell]

Would it be possible to set up an additional process that would move all raw images in their FIT format to the Amazon S3 bucket? We are already making use of the JPG move to S3.

We have a student project where additional access to the raw FIT files (as well as the processed JPG or TIFFs, and sky summaries) is important. The FIT files provide information on the exposure time/sensitivity. It also ensures they understand any steps that need to be taken for subsequent image processing. is there a wiki page for indi-allsky that discusses your management and use of all the different file formats, their locations on servers (under the various options available) and the lifetime of their availability (i.e. what policies are there for their lifetimes before deletion)? How long do you typically keep the raw FIT files before deletion? Thank you so much for all your work, Rick Gaitskell

[gaitskell]

I do see mention on the indi-allsky homepage of FIT files [optionally] being moved to the Amazon S3 bucket, but I haven't been able to find them in our actual implementation (only the JPG and the MP4 animations appear to be moved to S3). The option to create FIT has been turned on in the Config and I can see them on the local camera raspberry pi.

[aaronwmorris]

Sounds interesting!

I believe the the FITS images are saved as long as normal images, 30 days by default. Most people would probably run out of space given the size of those files if you actually saved that many. I could add a different policy for FITS if it becomes a problem.

I can work on syncing FITS (and "raw") images to S3 (and eventually GCP). I will probably make an optional checkbox to enable that feature.

[aaronwmorris]

I have merged #994 to upload FITS and RAW files to S3. I have not really tested the code yet, my lab environments are packed up at the moment since I am in the middle of a move.

[aaronwmorris]

As for the documentation regarding file storage, I can work on that over the next few days.

[gaitskell]

Aaron, Thank you. Updated to latest Main version.

I am seeing the FIT and TIF files in the Amazon S3, and can access them there.

The log is reporting this set of errors shown below on every cycle - and as a consequence (I believe) the UI reports the following Date: 2023-10-18 23:58:02 WARNING: Upload worker was restarted more than 10 times

Oct 18 20:44:09 PiAllSky [INFO] MainProcess-2167/MainThread allsky._queueManualTasks() #1005: Checking for manually submitted tasks
Oct 18 20:44:09 PiAllSky [INFO] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #493: Starting Upload-274 worker
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception: 
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception: pycurl.error: (3, '')
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception:     self.client.perform()
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception:   File "/home/admin/indi-allsky/indi_allsky/filetransfer/pycurl_sftp.py", line 131, in put
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception:     raise e from e
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception:   File "/home/admin/indi-allsky/indi_allsky/filetransfer/pycurl_sftp.py", line 152, in put
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception:     response = client.put(**put_kwargs)
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception:   File "/home/admin/indi-allsky/indi_allsky/uploader.py", line 366, in processUpload
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception:     self.processUpload(u_dict)
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception:   File "/home/admin/indi-allsky/indi_allsky/uploader.py", line 127, in saferun
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception:     self.saferun()
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception:   File "/home/admin/indi-allsky/indi_allsky/uploader.py", line 99, in run
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception: Traceback (most recent call last):
Oct 18 20:44:09 PiAllSky [INFO] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #493: Starting Upload-273 worker
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception: 
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception: pycurl.error: (3, '')
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception:     self.client.perform()
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception:   File "/home/admin/indi-allsky/indi_allsky/filetransfer/pycurl_sftp.py", line 131, in put
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception:     raise e from e
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception:   File "/home/admin/indi-allsky/indi_allsky/filetransfer/pycurl_sftp.py", line 152, in put
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception:     response = client.put(**put_kwargs)
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception:   File "/home/admin/indi-allsky/indi_allsky/uploader.py", line 366, in processUpload
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception:     self.processUpload(u_dict)
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception:   File "/home/admin/indi-allsky/indi_allsky/uploader.py", line 127, in saferun
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception:     self.saferun()
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception:   File "/home/admin/indi-allsky/indi_allsky/uploader.py", line 99, in run
Oct 18 20:44:09 PiAllSky [ERROR] MainProcess-2167/MainThread allsky._fileUploadWorkerStart() #486: Upload worker exception: Traceback (most recent call last):
Oct 18 20:44:07 PiAllSky [INFO] Capture-1-2171/MainThread indi.getTelescopeRaDec() #842: Telescope Coord: RA 21.80, Dec 41.74
Oct 18 20:44:07 PiAllSky [WARNING] Capture-1-2171/MainThread indi.getCcdTemperature() #852: Sensor temperature not supported
Oct 18 20:44:07 PiAllSky [INFO] Capture-1-2171/MainThread capture.detectMoonMode() #1290: Moon altitude: -6:44:25.6, phase 18.5%
Oct 18 20:44:07 PiAllSky [INFO] Capture-1-2171/MainThread capture.detectNight() #1272: Sun altitude: -31:14:54.2
Oct 18 20:44:07 PiAllSky [INFO] Capture-1-2171/MainThread capture.saferun() #210: Exposure state: BUSY
Oct 18 20:44:07 PiAllSky [INFO] Capture-1-2171/MainThread capture.saferun() #209: Camera last ready: 6.0s
Oct 18 20:44:04 PiAllSky [INFO] MainProcess-2167/Upload-271 pycurl_sftp.put() #118: pycurl URL: sftp://:22/allsky/latest_metadata.json
Oct 18 20:44:04 PiAllSky [INFO] MainProcess-2167/Upload-271 generic.put() #50: Uploading /tmp/tmp05f6je4f.json

[aaronwmorris]

Curl error code 3 is a Malformed URL: https://curl.se/libcurl/c/libcurl-errors.html

That exception is coming from the file transfer code, not the S3 code. From the error, it looks like you have SFTP file uploads enabled, but the hostname entry is blank, leading to the malformed URL sftp://:22/allsky/latest_metadata.json I am guessing you are not really using the file transfer and probably just need to disable that.

[gaitskell]

Able to work with FITS files direct from S3 servers in a Python Notebook / Google Colab Sessions - this is fantastic - thank you

Example python code below (note some other pip installs for AWS+relaunching Colab kernel, AWS authentication required)

from PIL import Image
from astropy.io import fits
# Requires install s3fs
url = "s3://cfpu-allskycamera-test02/"+file
with fits.open(url, use_fsspec=True, fsspec_kwargs={"anon": True}) as hdul:  
  cutout = hdul[0].section[100:900, 300:1100]
  print( cutout.shape )
Image.fromarray(cutout)

Details from FITS file SIMPLE = T / file does conform to FITS standard
BITPIX = 16 / number of bits per data pixel
NAXIS = 2 / number of data axes
NAXIS1 = 1392 / length of data axis 1
NAXIS2 = 1040 / length of data axis 2
EXTEND = T / FITS dataset may contain extensions
COMMENT FITS (Flexible Image Transport System) format is defined in 'Astronomy COMMENT and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H BSCALE = 1 / default scaling factor
BZERO = 32768 / offset data range to that of unsigned short
ROWORDER= 'TOP-DOWN' / Row Order
INSTRUME= 'indi-allsky' / CCD Name
TELESCOP= 'indi-allsky' / Telescope name
EXPTIME = 1.244809E+01 / Total Exposure Time (s)
PIXSIZE1= 6.449219E+00 / Pixel Size 1 (microns)
PIXSIZE2= 6.449219E+00 / Pixel Size 2 (microns)
XBINNING= 1 / Binning factor in width
YBINNING= 1 / Binning factor in height
XPIXSZ = 6.449219E+00 / X binned pixel size in microns
YPIXSZ = 6.449219E+00 / Y binned pixel size in microns
FRAME = 'Light ' / Frame Type
IMAGETYP= 'Light Frame' / Frame Type
FOCALLEN= 1.000E+01 / Focal Length (mm)
APTDIA = 1.000E+01 / Telescope diameter (mm)
SCALE = 1.330474E+02 / arcsecs per pixel
SITELAT = 4.174070E+01 / Latitude of the imaging site in degrees
SITELONG= -7.130860E+01 / Longitude of the imaging site in degrees
AIRMASS = 1.000000E+00 / Airmass
OBJCTAZ = 2.700008E+02 / Azimuth of center of image in Degrees
OBJCTALT= 8.999762E+01 / Altitude of center of image in Degrees
OBJCTRA = '21 02 48.43' / Object J2000 RA in Hours
OBJCTDEC= '41 38 26.80' / Object J2000 DEC in Degrees
RA = 3.157018E+02 / Object J2000 RA in Degrees
DEC = 4.164078E+01 / Object J2000 DEC in Degrees
PIERSIDE= 'EAST ' / East, looking West
EQUINOX = 2000 / Equinox
DATE-OBS= '2023-10-18T23:59:51.956' / UTC start date of observation
OBJECT = 'All Sky '
OBSERVER= 'Gaitskell, Richard'
SITE = 'Brown'
NOTES = 'Early test runs'
COMMENT Generated by INDI

[gaitskell]

You are correct concerning cause of LOG Errors. I had turned on the Config>File Transfer of FITS as well as the Config>Object Storage of the FITs

[aaronwmorris]

Closing this for now. Let me know if there is something else I can do.