search

Wanchai / FTPbucket

FTPbucket is a PHP script that enables you to sync your BitBucket or GitHub repository with any web-server
102 stars 22 forks source link

Goes into an infinite loop if the script takes too long #27

Open Wanchai opened 5 years ago

Wanchai commented 5 years ago

Since merging PR #24, I tried to add a folder containing ~700 files and the proccess started all over after file 141: 22.08.2019, 00:24:44: Script called for BitBucket JSON payload 22.08.2019, 00:24:44: Received a push from {###} 22.08.2019, 00:24:44: Modifications detected on the branch {master} 22.08.2019, 00:24:44: Commencing transfer for change 1 22.08.2019, 00:24:46: 703 files changed. 22.08.2019, 00:24:46: Created new directory fol2 copy 22.08.2019, 00:24:47: Uploaded: ../bb/fol2 copy/001.txt 22.08.2019, 00:24:47: Uploaded: ../bb/fol2 copy/002.txt 22.08.2019, 00:24:48: Uploaded: ../bb/fol2 copy/003.txt 22.08.2019, 00:24:48: Uploaded: ../bb/fol2 copy/004.txt [...] 22.08.2019, 00:25:43: Uploaded: ../bb/fol2 copy/140.txt 22.08.2019, 00:25:44: Uploaded: ../bb/fol2 copy/141.txt 22.08.2019, 00:25:44: Script called for BitBucket JSON payload 22.08.2019, 00:25:44: Received a push from {###} 22.08.2019, 00:25:44: Modifications detected on the branch {master} 22.08.2019, 00:25:44: Commencing transfer for change 1 22.08.2019, 00:25:44: Uploaded: ../bb/fol2 copy/142.txt 22.08.2019, 00:25:45: Uploaded: ../bb/fol2 copy/143.txt 22.08.2019, 00:25:45: Uploaded: ../bb/fol2 copy/144.txt 22.08.2019, 00:25:45: Uploaded: ../bb/fol2 copy/145.txt 22.08.2019, 00:25:46: Uploaded: ../bb/fol2 copy/146.txt 22.08.2019, 00:25:46: 703 files changed. 22.08.2019, 00:25:46: Uploaded: ../bb/fol2 copy/147.txt 22.08.2019, 00:25:46: Uploaded: ../bb/fol2 copy/001.txt 22.08.2019, 00:25:47: Uploaded: ../bb/fol2 copy/148.txt 22.08.2019, 00:25:47: Uploaded: ../bb/fol2 copy/002.txt

d2roth commented 5 years ago

Did you fix this? It looks like two pushes are running at the same time. I know working with this script we always needed to wait before doing a second push because if we didn't we would have unpredictable results.

d2roth commented 5 years ago

@Wanchai so I think this is a case of handle_rate_limit coming into play. Basically when a 429 HTTP status code is hit the rate limiter stops execution for a bit so BitBucket isn't hammered.

Did both of your pushes affect the same files?

I just did a test with about 4k files and this is what the results were: lgofile.txt

05.09.2019, 13:10:20: Script called for BitBucket JSON payload
05.09.2019, 13:10:20: Received a push from {...}
05.09.2019, 13:10:20: Modifications detected on the branch {master}
05.09.2019, 13:10:20: Commencing transfer for change 1
05.09.2019, 13:10:57: 3996 files changed.
05.09.2019, 13:10:57: Uploaded: /public_html/bitbucket-v2-ftpbucket-test/test-1-1.txt
05.09.2019, 13:10:57: Uploaded: /public_html/bitbucket-v2-ftpbucket-test/test-1-10.txt
05.09.2019, 13:10:58: Uploaded: /public_html/bitbucket-v2-ftpbucket-test/test-1-100.txt
...
05.09.2019, 14:00:53: Uploaded: /public_html/bitbucket-v2-ftpbucket-test/test-2-846.txt
05.09.2019, 14:00:53: Uploaded: /public_html/bitbucket-v2-ftpbucket-test/test-2-847.txt
05.09.2019, 14:00:53: Uploaded: /public_html/bitbucket-v2-ftpbucket-test/test-2-848.txt
05.09.2019, 14:00:53: Uploaded: /public_html/bitbucket-v2-ftpbucket-test/test-2-849.txt
05.09.2019, 14:04:55: Uploaded: /public_html/bitbucket-v2-ftpbucket-test/test-2-85.txt
05.09.2019, 14:04:55: Uploaded: /public_html/bitbucket-v2-ftpbucket-test/test-2-850.txt
05.09.2019, 14:04:56: Uploaded: /public_html/bitbucket-v2-ftpbucket-test/test-2-851.txt
05.09.2019, 14:04:56: Uploaded: /public_html/bitbucket-v2-ftpbucket-test/test-2-852.txt
05.09.2019, 14:04:56: Uploaded: /public_html/bitbucket-v2-ftpbucket-test/test-2-853.txt
05.09.2019, 14:04:56: Uploaded: /public_html/bitbucket-v2-ftpbucket-test/test-2-854.txt
05.09.2019, 14:04:57: Uploaded: /public_html/bitbucket-v2-ftpbucket-test/test-2-855.txt
05.09.2019, 14:04:58: Uploaded: /public_html/bitbucket-v2-ftpbucket-test/test-2-856.txt
05.09.2019, 14:04:59: Uploaded: /public_html/bitbucket-v2-ftpbucket-test/test-2-857.txt
...
05.09.2019, 16:11:48: Uploaded: /public_html/bitbucket-v2-ftpbucket-test/test-4-996.txt
05.09.2019, 16:11:48: Uploaded: /public_html/bitbucket-v2-ftpbucket-test/test-4-997.txt
05.09.2019, 16:11:48: Uploaded: /public_html/bitbucket-v2-ftpbucket-test/test-4-998.txt
05.09.2019, 16:11:49: Uploaded: /public_html/bitbucket-v2-ftpbucket-test/test-4-999.txt
05.09.2019, 16:11:49: Transfer done for branch {master}
05.09.2019, 16:11:49: Push completed. There were {4052} total CURL calls made.

logconnection.txt

A PUSH was started at 05.09.2019, 13:10:20
The push finally finished at 05.09.2019, 16:11:49.
In total it took 10889 seconds.

So basically I think what was happening for you might have been that you started the push and then it was still running because there is no end "Push completed" line in your sample.

Does that match your findings?