search

cubewise-code / tm1py

TM1py is a Python package that wraps the TM1 REST API in a simple to use library.
http://tm1py.readthedocs.io/en/latest/
MIT License
190 stars 110 forks source link

implement mpu file upload #1190

Closed MariusWirtz closed 2 weeks ago

MariusWirtz commented 2 weeks ago

Fixes TM1 v12 (in PA aaS) constraint of 200 MB max file size in file update / create endpoints

Speeds up file upload through parallelization of multi-part upload.

MariusWirtz commented 2 weeks ago

Fixes #1186

github-actions[bot] commented 2 weeks ago

Tests completed for environment: tm1-11-cloud. Check artifacts for details.

Hubert-Heijkers commented 2 weeks ago

Nice one @MariusWirtz!

Quick question though, just because I don't know enough of the details, what happens if one of the requests, specifically the POST of a part, fails? Does the underlying POST repeat that request until it doesn't?

Asking because MPU, other then speeding up things, also helps deal with issues of uploading large files which could still happen whilst uploading a part. Even if the upload of such part succeeds but, due to some network glitch, that is never returned to the initiator of that request. You don't need to clean up extra parts or anything, I'll do that for you in the end, but you might need to retry uploading a part in some of these cases???

PS There is absolutely no constraint on the TM1 v12 side that limits the maximum file size, this is something that's imposed by the PA SaaS environment apparently (just saying;-).

MariusWirtz commented 2 weeks ago

Good catch @Hubert-Heijkers. Thank you

I will make sure to include a retry on the part uploads.

github-actions[bot] commented 2 weeks ago

Tests completed for environment: tm1-11-cloud. Check artifacts for details.