CABLE-LSM / meorg_client

API Client for ModelEvaluation.org.
Apache License 2.0
0 stars 1 forks source link

Revised file-upload logic. #24

Closed bschroeter closed 1 month ago

bschroeter commented 1 month ago

Due to the API changes from ME.org, the file upload logic needed changing.

Summary:

Originally, we thought that we could upload everything in a directory at once, however, the receiving server has a packet limit, which we have had raised to 200mb. This will allow large (but singular) files to be transferred up to this size. Smaller files can feasibly be sent all at once.

bschroeter commented 1 month ago

@SeanBryan51, New commit added with the suggested fixes. I think we're in a good spot now, it's much cleaner.

bschroeter commented 1 month ago

I've added some comments regarding the upload_files function. The design of upload_files is bugging me a bit. Would it make more sense to have a separate function for uploading in memory file objects? This way I don't think you would need a temporary _files variable. Sorry I probably should have raised this in my first review.

A singular function is implemented. It would be basically a heap of replicated code to handle objects in a separate function. The use case on objects is slim, mostly for testing and direct connection via Python. 99% of the usage will be on the command line with a path that gets opened.