search

cyverse / gocommands

iRODS Command-line Tools written in Go
Other
29 stars 18 forks source link

Parallel upload creates stale replicas #14

Closed iychoi closed 1 year ago

iychoi commented 1 year ago

Hi Illyoung, I finally found the time over the last days to do some more trials.

Here my observation:

11072023_put_tests.txt: Transferred files in the range of 10-100GB (10GB steps, ) Out of 32 transfers, two ended up with stale replicas Both stale replicas where for test transfers of quite big files (80GB + 90GB) 12072023_put_tests.txt Transferred files in the range 70-100GB (three times each) All ended up in a 'good' state, no 'stale' replicas 13072023_put_tests.txt Again 10-100GB in 10GB steps One 'stale' replica. This time not for a big file, instead for the smallest (10GB) file that was tested Also the 100GB file is in a 'locked' state, but that was likely due to a connection loss during the transfer and is not relevant Small files: Also uploaded ~15k of small 1MB No files in 'stale' state detected for all transfers

These are my observations with v0.7.4 so far. As you can see, it seems that it still happens occasionally that files are marked as 'stale', but I have no idea why.

iychoi commented 1 year ago

This might be due to wrong implementation of replica_close API. Fixed in v0.7.6. But need confirmation.

https://github.com/cyverse/gocommands/releases/tag/v0.7.6

iychoi commented 1 year ago

Jelte confirmed that the new version works.

Hello Illyoung,

I did finally test v0.7.6.

In short: The issue seems to be fixed!

I so far uploaded again single files from 10-100GB --> No stale replicas detected!

I also tried with quite a lot of smaller files (1MB) --> No stale replicas detected!

I just tested the 'put' command, but everything seems great so far. Thanks again!

Best, Jelte