Wanchai / FTPbucket

FTPbucket is a PHP script that enables you to sync your BitBucket or GitHub repository with any web-server
102 stars 22 forks source link

Bitbucket Webhook Timeout #12

Closed carlzata closed 7 years ago

carlzata commented 7 years ago

Hi @Wanchai,

Thank you for this awesome script! It works perfectly when i try to push few files, but the webhook fails if webhook runtime is more than 10seconds which i think Bitbucket wont fix. Is there anyway we can fix these timeouts?

Thanks!

d2roth commented 7 years ago

Hey @carlzata, I ran into that issue and made some modifications to this script. This was only tested on our servers and my local environment but it does the trick for us.

Some improvements that should be done but I haven't had the time to do include, checking if the ftp connection succeeds before closing the connection so in BitBucket/GitHub you at least know that the connection is fine in the hook. Also there is no way of knowing if a deployment is ongoing except if you read the logfile.txt. So I don't know what would happen if 2 repositories got pushed to at similar times.

Anyways without further ado here is my solution:

Basically I used ignore_user_abort(true). I believe this will work with FTPBucket in its current state but I have made other changes in my copy so you might need to play around with it.

To track how long each push takes I created a new file called connection_log.txt.

Add this to the top of deploy.php

<?php
/*
 * Set the memory limit high so that we don't hit memory errors.
 */
ini_set('memory_limit', '250M');

/*
 * We don't want execution errors so we will just set unlimited execution time.
 */
ini_set('max_execution_time', 0);

ob_end_clean();

ignore_user_abort(true);
set_time_limit(0);

/* 
 * We are going to forcefully close the connection which allows
 * BitBucket/GitHub to move on and let our script run uninterrupted.
 */
header("Connection: close\r\n");
header("Content-Encoding: none\r\n");

/* Start output buffering so we can get a content size */
ob_start();
if ($fp = fopen('connection_log.txt', 'a')) {
    $start_time = microtime(true);
    fwrite($fp, 'A PUSH was started at ' . $start_time . "" . PHP_EOL);
}
echo "Roger that! The deployment script has received your request.";

$size = ob_get_length();

header("Content-Length: $size");

/*
 * Strange behaviour, will not work
 * Unless both are called !
 */
ob_end_flush();
flush();

/* Now clean out */
ob_end_clean();

Add this to the bottom of deploy.php

$end_time = microtime(true);
fwrite($fp, 'The push finally finished at ' . $end_time . "." . PHP_EOL);
fwrite($fp, 'In total it took ' . ($end_time - $start_time). " seconds." . PHP_EOL . PHP_EOL);
fclose($fp);
carlzata commented 7 years ago

Hi @d2roth

Thanks for your help, timeout is gone. Though checking the deployment logs, the process still stops at some time and does not finish the deployments.

d2roth commented 7 years ago

Hmm... 2 questions to start debugging, What does connection_log.txt say and What does the logfile.txt say? Connection_log.txt should tell you the length that the deployment took if it completes. Logfile.txt will tell you the last file that was pushed. I am wondering if maybe you might be hitting a timeout set by the server or maybe a disk quota or something similar.

Also what does BitBucket say? Does it say "Roger that! The deployment script has received your request." or does it give an error?

Just out of curiosity how big is the push you are trying to make? Like # of files and total file size.

When I was testing I did a push of about 120mb and 26,000 files.

carlzata commented 7 years ago

Hi @d2roth

Connection_log.txt only contains this "A PUSH was started at 1477760241.35". I think i might be hitting server timeout with the script, but yeah i get the text "Roger that! The deployment script has received your request."

And the files included in the push doesnt seem to be that big compare to your test, it contains around 900+ files only around 10mb.

d2roth commented 7 years ago

Hey @carlzata, Hmm... You could have a look at phpinfo() and see what your timeout is. I am working with a VPS so I can change things but if you are on a shared host you probably are hitting a timeout.

Depending on the host sometimes you can modify the timeout with a .htaccess file instead of in the script like I did. Here are some different ways to change the timeout: http://www.thecave.info/override-the-default-php-execution-timeout-limit/

I don't know what else you could try unless you can get a dedicated box for just running the deployment script.

Wanchai commented 7 years ago

Hey guys, Thanks @d2roth for your contribution. I haven't tested large pushes for a long time. I implemented your code with a few changes.

@carlzata Did you try to go to the admin page and check your logs? The logfile should contains informations about the transfer. Do you have a 'transfer done' message?

carlzata commented 7 years ago

@Wanchai i still get timeouts during deployments, but i don't think the script is to be blamed but the restrictions from the server.

Anyway thanks for this awesome tool you have, i will close this ticket now