Currently cannot fetch from hitran.org

[erwanp]

I'll retry in a few days

[suzil]

Would we reduce the load to HITRAN if there was a persistent data store on the Lambda runtime environment? I think RADIS caches this data right?

[erwanp]

Yes, we could only download once every few months. HITRAN database doesn't change much. Total data is ~2 GB.

Radis-Lab already caches everything, using Radis's "fetch_hitran" : https://github.com/radis/radis-lab/blob/main/databases/download_hitran.py

[suzil]

I might try attaching EFS to give the Lambdas a persistent file system in order to cut down on downloads. I'll have to check on the costs of this.

I'm also checking on possibly moving this to something other than Lambda but Lambda is so much cheaper of an option. The hosting costs of the current setup is ~$1/month.

[suzil]

Possible approaches to this:

• Cache the HITRAN data on EFS
• Move the data to a database that we can control, preferably a serverless kind that charges based on usage such as Aurora Serverless or DynamoDB On-Demand

EFS is probably simpler since it's just a file system, but I'd really like to keep costs low for whatever approach is used (low meaning less than a few dollars a month). RADIS app isn't used so frequently that if we use a data layer that prices per request, costs tend to be cheap.

[suzil]

Is this still relevant now that we have a persistent server and run the download_hitran.py script? @erwanp https://github.com/suzil/radis-app/blob/main/server/download_hitran.py

[erwanp]

Not an issue anymore !

[erwanp]

Btw @suzil , can users download from the persistent server ?

I.e., it could be a very easy way to directly get the Radis-formatted HDF5 files for Hitran, or Hitemp, and save a lot of the parsing time I mentioned few weeks ago

[suzil]

There's no reason we couldn't serve users data that is stored on the server.