Wauplin
commented
1 year ago Hi @narugo1992, thanks for raising the question. I'm not sure yet how/if we want to solve this issue at the moment, given the particular use case you have. In general concurrent writes in git are quite difficult to tackle as they might be changing the same resource and we don't want to loose some information in the process.
In your particular case it seems quite straightforward to do a try/except around the upload_file call and implement the backoff strategy on your side. This gives you flexibility (how many attempts? how long to wait? on which error? ...). A good thing is that if you are committing LFS files, you will not have to re-upload them at each attempt.
# /!\ untested and quite simple implementation
def upload_with_retries(...):
attempt = 0
while True:
attempt += 1
try:
return upload_file(...)
except HfHubHTTPError as error:
if error.status_code != 412 or attempt >= MAX_ATTEMPTS:
raise
time.sleep(1)Also just so you know, it's possible to upload/delete several files at the same time using the low level create_commit method. That might solve your concurrency issue (you can upload tens of files in the same commit).
Note: I think the more general question you are raising is the ability to update mulitple files concurrently or on the fly, the same way you would deal with a local filesystem or a S3 bucket. This is currently not so easy because of the underlying git-based backend -and will most probably still be the case at short/mid-term. cc @Pierrci @julien-c about recent discussions to "hide" more the git constraints
narugo1992
Is your feature request related to a problem? Please describe. Here are 412 errors when I tried to upload images to dataset in multiple parallel runners: https://github.com/narugo1992/gchar/actions/runs/4599743391/jobs/8125512962#step:14:272
This error is most likely caused by conflicts between multiple submissions that occur simultaneously during multi-threaded uploading.
Describe the solution you'd like Regarding this issue, the solution I can think of is: when 412 errors are detected due to multi-threaded uploading, automatic retry can be performed (of course, considering that retrying is not always appropriate in every case, this feature can be made optional).
Describe alternatives you've considered If a specific type of exception can be thrown when such errors occur, and a manual function for refreshing and retrying is provided, then the user can also control whether to retry or not. This solution can also be an alternative to the above-mentioned solution.
Additional context This scenario is common when dealing with a large number of files with a large overall size, and the data needs to be automatically updated based on the timed execution functionality provided by online platforms such as Github Action. Since online runners cannot provide very large hard disk space, resources can only be generated and uploaded at the same time to reduce the cost of hard disk space. Under this circumstance, if stable concurrent uploading or processing (such as deleting) of data in the dataset can be achieved, it will greatly facilitate the deployment of automatic update functionality for the dataset.