leonstafford
commented
4 years ago @crstauf looks good, I'll give it a try now
Closed crstauf closed 1 month ago
leonstafford
commented
4 years ago @crstauf looks good, I'll give it a try now
crstauf
commented
4 years ago @leonstafford I've completed the BitBucket progress indicators: I changed it to be one-line status updates, and the file processing is a progress bar.
Also includes some debug commands and warnings, and a fix for a PHP notice.
Note that I do not have a site that results in multiple batches, so if you can deployment that uses multiple batches that'd be great!
leonstafford
commented
4 years ago @crstauf awesome, trying out now!
leonstafford
commented
4 years ago Maybe still a bit of tweak needed, this is what I get with batch size set to 1 (Advanced tab):
Generating file list [0/17051]
Preparing for export
Processing URLs 47 / 47 100% [============================================================] 0:06 / 0:11
Performing post process actions
Success: Generated static site archive in 6.842641 seconds
x230$ wp statichtmloutput deploy
Deploying static site
Bootstrapping completed
Archive loaded
Preparing for deployment
Batch 1: processing files 1 / 41 100% [===================================================] 0:00 / 0:00
Batch 1: sending 1 files
Batch 1: sent 1 files
Batch 2: processing files 1 / 40 100% [===================================================] 0:00 / 0:00
Batch 2: sending 1 files
Batch 2: sent 1 files
Batch 3: processing files 1 / 39 100% [===================================================] 0:00 / 0:00
Batch 4: processing files 1 / 38 100% [===================================================] 0:00 / 0:00
Batch 5: processing files 1 / 37 100% [===================================================] 0:00 / 0:00
Batch 5: sending 1 files
Batch 5: sent 1 files
Batch 6: processing files 1 / 36 100% [===================================================] 0:00 / 0:00
Batch 7: processing files 1 / 35 100% [===================================================] 0:00 / 0:00
Batch 7: sending 1 files
Batch 7: sent 1 files
Batch 8: processing files 1 / 34 100% [===================================================] 0:00 / 0:00
Batch 8: sending 1 files
Batch 8: sent 1 files
Batch 9: processing files 1 / 33 100% [===================================================] 0:00 / 0:00
Batch 9: sending 1 files
Batch 9: sent 1 files
Batch 10: processing files 1 / 32 100% [==================================================] 0:00 / 0:00
Batch 10: sending 1 files
Batch 10: sent 1 files
Batch 11: processing files 1 / 31 100% [==================================================] 0:00 / 0:00
Batch 11: sending 1 files
Batch 11: sent 1 files
Batch 12: processing files 1 / 30 100% [==================================================] 0:00 / 0:00
Batch 13: processing files 1 / 29 100% [==================================================] 0:00 / 0:00
Batch 13: sending 1 files
Batch 13: sent 1 files
Batch 14: processing files 1 / 28 100% [==================================================] 0:00 / 0:00
Batch 14: sending 1 files
Batch 14: sent 1 files
Batch 15: processing files 1 / 27 100% [==================================================] 0:00 / 0:00
Batch 15: sending 1 files
Batch 15: sent 1 files
Batch 16: processing files 1 / 26 100% [==================================================] 0:00 / 0:00
Batch 16: sending 1 files
Batch 16: sent 1 files
Batch 17: processing files 1 / 25 100% [==================================================] 0:00 / 0:00
Batch 18: processing files 1 / 24 100% [==================================================] 0:00 / 0:00
Batch 19: processing files 1 / 23 100% [==================================================] 0:00 / 0:00
Batch 20: processing files 1 / 22 100% [==================================================] 0:00 / 0:00
Batch 21: processing files 1 / 21 100% [==================================================] 0:00 / 0:00
Batch 22: processing files 1 / 20 100% [==================================================] 0:00 / 0:00
Batch 23: processing files 1 / 19 100% [==================================================] 0:00 / 0:00
Batch 23: sending 1 files
Batch 23: sent 1 files
Batch 24: processing files 1 / 18 100% [==================================================] 0:00 / 0:00
Batch 25: processing files 1 / 17 100% [==================================================] 0:00 / 0:00
Batch 26: processing files 1 / 16 100% [==================================================] 0:00 / 0:00
Batch 27: processing files 1 / 15 100% [==================================================] 0:00 / 0:00
Batch 28: processing files 1 / 14 100% [==================================================] 0:00 / 0:00
Batch 29: processing files 1 / 13 100% [==================================================] 0:00 / 0:00
Batch 30: processing files 1 / 12 100% [==================================================] 0:00 / 0:00
Batch 31: processing files 1 / 11 100% [==================================================] 0:00 / 0:00
Batch 32: processing files 1 / 10 100% [==================================================] 0:00 / 0:00
Batch 33: processing files 1 / 9 100% [===================================================] 0:00 / 0:00
Batch 34: processing files 1 / 8 100% [===================================================] 0:00 / 0:00
Batch 35: processing files 1 / 7 100% [===================================================] 0:00 / 0:00
Batch 36: processing files 1 / 6 100% [===================================================] 0:00 / 0:00
Batch 37: processing files 1 / 5 100% [===================================================] 0:00 / 0:00
Batch 38: processing files 1 / 4 100% [===================================================] 0:00 / 0:00
Batch 39: processing files 1 / 3 100% [===================================================] 0:00 / 0:00
Batch 40: processing files 1 / 2 100% [===================================================] 0:00 / 0:00
Batch 41: processing files 1 / 1 100% [===================================================] 0:00 / 0:00
Deployed to: bitbucket in 00:00:29Where crawling took just 1 line of output, each of those are what deploy used.
Let me try again with larger batch size for comparison.
leonstafford
commented
4 years ago with batch size of 10 and what looks like 41 files to deploy:
x230$ wp statichtmloutput generate
Generating file list
Preparing for export
Processing URLs 47 / 47 100% [==================================================================================================================================================================================================] 0:07 / 0:13
Performing post process actions
Success: Generated static site archive in 6.855561 seconds
x230$ wp statichtmloutput deploy
Deploying static site
Bootstrapping completed
Archive loaded
Preparing for deployment
Batch 1: processing files 10 / 41 100% [========================================================================================================================================================================================] 0:00 / 0:00
Batch 1: sending 10 files
Batch 1: sent 10 files
Batch 2: processing files 10 / 31 100% [========================================================================================================================================================================================] 0:00 / 0:00
Batch 2: sending 10 files
Batch 2: sent 10 files
Batch 3: processing files 10 / 21 100% [========================================================================================================================================================================================] 0:00 / 0:00
Batch 3: sending 10 files
Batch 3: sent 10 files
Batch 4: processing files 10 / 11 100% [========================================================================================================================================================================================] 0:00 / 0:00
Batch 4: sending 10 files
Batch 4: sent 10 files
Batch 5: processing files 1 / 1 100% [==========================================================================================================================================================================================] 0:00 / 0:00
Batch 5: sending 1 files
Batch 5: sent 1 files
Deployed to: bitbucket in 00:00:20
x230$ @crstauf for the CLI, we can more easily use the total number of files to deploy before the process starts, with DeployQueue::getTotal();, vs the UI, where I'm just outputting how many more are remaining in DeployQueue each tick.
If we query the deployBatchSize option, then we can also predict the number of batches (each batch may not need to transfer some or any files, depending on deploy cache, but should allow for consistent progress indicator)
leonstafford
commented
4 years ago @crstauf this may help generate bigger batches for testing: https://developer.wordpress.org/cli/commands/post/generate/
crstauf
commented
4 years ago @leonstafford I think you'll be much more pleased with 59c2124's handling of batches.
The label of the progress bar switches between Processing files x / y and Uploading batch x / y. Because of how quickly files are processed, you may only see Uploading batch x / y: I had to generate several posts (thanks for the CLI command!) and set batch size to 50+ to see the switch occur.
leonstafford
commented
4 years ago @crstauf awesome, giving it a go now!
leonstafford
commented
4 years ago @crstauf looking pretty good!
Sorry to find one issue - if batch size is larger than files needing to be deployed, it goes straight to 100% and hangs while deploying. Actually, looks the same on the smaller batch sizes - the % increase is recorded before the batch has confirmed successful.
(ps. did you use the deploy_cache delete? maybe why it's so fast ;)
x230$ wp statichtmloutput deploy_cache delete
no --force given. Please type 'yes' to confirm deletion of Deploy Cache
yes
Success: Deleted Deploy Cache
x230$ wp statichtmloutput deploy
Deploying static site
Bootstrapping completed
Archive loaded
Preparing for deployment
Uploading batch 41 / 41 100% [========================================================================================================================================================================] 1:53 / 1:16
Deployed to: bitbucket in 00:01:52
x230$ wp statichtmloutput deploy_cache delete
no --force given. Please type 'yes' to confirm deletion of Deploy Cache
yes
Success: Deleted Deploy Cache
x230$ wp statichtmloutput options set deployBatchSize 100
x230$ wp statichtmloutput deploy
Deploying static site
Bootstrapping completed
Archive loaded
Preparing for deployment
Uploading batch 1 / 1 100% [==========================================================================================================================================================================] 0:00 / 0:00@crstauf ps. BitBucket/GitLab are the hardest ones due to batching, others will be easier :D
crstauf
commented
4 years ago if batch size is larger than files needing to be deployed, it goes straight to 100% and hangs while deploying
@leonstafford Yes, this is correct: currently, the actual upload is not included in the progress bar, just the processing of the files. This was done because initially I was thinking there was not a decent way to track batches, but I think by adding the total number of batches onto the progress bar step count, that might work. I'll try that next.
did you use
deploy_cache delete
Yes I did, still really fast for me.
crstauf
commented
4 years ago @leonstafford Okay, I think ba04d72 is it. :smile:
leonstafford
commented
4 years ago @crstauf all looking great!
sending batch 1 / 1 97 % [= from beginning then long pause. crstauf
commented
4 years ago all except for large batch size
@leonstafford This is because the files are processed quickly, and the long pause is for the actual deployment. What would you like to or are expecting to see?
leonstafford
commented
4 years ago @crstauf yep, you're right - let's lock this one in!
leonstafford
commented
4 years ago @crstauf - do you want me to merge this one in now or add in the other deploy methods to the PR? I'll probably be working on Windows support in another branch, so shouldn't be much new in master to rebase this branch on
crstauf
commented
4 years ago @leonstafford I think I’d like to do the others as well. Client work has picked up a little bit, so I’ve not as much OSS dev time as I had, but it’s still on my radar and I hope to get at it again today or tomorrow.
leonstafford
commented
4 years ago Hi @crstauf I kinda disappeared for a few weeks, but starting to open the overflowing inbox and looking at code again now. I'll be looking to wrap up the Windows compatibility and v6 to v7 migration/file cleanup work, so we can get this progress stuff in anytime. Hope all is going well with you and look forward to chatting more as things settle down.
crstauf
commented
4 years ago @leonstafford Same: paying work has shot upwards. If you want to merge BitBucket progress bar in, I'll swing around to the others in time.
leonstafford
commented
4 years ago @crstauf sounds like a great problem to have 🎈 No worries, I'll ping you on this issue when there's any movement
Closes #101.