application upload - perf improvement using joblib

[maburlik]

Fixes Bug 8657234: sfctl application upload has perf issue since it uploads files one at a time. 7.2 release validation timed out and required timeout increase for small app

Changes include:

• Updated 'application upload' native imagestore application upload to parallelize uploads, greatly reducing overall upload time for applications with many smaller files.
• Improved progress log shown by flag '--show-progress' to have progress slider, using tqdm module.

Verified:

• [x] Build and test CI passes
• [x] History and readme updated to reflect changes
• [ ] N/A Package version updated according to semantic versioning rules
• [x] Tests modified or added, when applicable
• [ ] N/A Updated code owners file, when applicable
• [x] Read the PR checklist

During 7.2 release validation the app uploads timed out at 5 min for some small apps, which led me to look at the implementation of application upload. Since it takes time to start file transfers for each file it's commonly beneficial to do this across multiple threads/processes.

After the change to use machine no. threads for degree of parallelism, the app that was timing out uploading at 5 min was able to be uploaded in 10 sec.

From src/test/current/scripts/linuxonebox/installgithubsamples.sh:

Uploading path: CounterServiceWebServicePkg/Code/runtimes/debian-x64: 0it [00:00, ?it/s] [114/342] files, CounterServiceWebServicePkg/Code/runtimes/debian-x64/.dir Time left: 1193 seconds Uploading path: CounterServiceWebServicePkg/Code/runtimes/debian-x64/native: 100%|█| 1/1 [00:00<00:00, 7.1 [116/342] files, CounterServiceWebServicePkg/Code/runtimes/debian-x64/native/.dir Time left: 1193 seconds Uploading path: CounterServiceWebServicePkg/Code/refs: 100%|█████████████| 158/158 [00:03<00:00, 45.96it/s] [275/342] files, CounterServiceWebServicePkg/Code/refs/.dir Time left: 1189 seconds Uploading path: CounterServiceWebServicePkg/Data: 100%|██████████████████████| 1/1 [00:00<00:00, 9.33it/s] [277/342] files, CounterServiceWebServicePkg/Data/.dir Time left: 1189 seconds Uploading path: CounterServiceWebServicePkg/Config: 100%|████████████████████| 2/2 [00:00<00:00, 17.54it/s] [280/342] files, CounterServiceWebServicePkg/Config/.dir Time left: 1189 seconds Uploading path: CounterServicePkg: 100%|█████████████████████████████████████| 3/3 [00:00<00:00, 26.66it/s] [284/342] files, CounterServicePkg/.dir Time left: 1189 seconds Uploading path: CounterServicePkg/Code: 100%|████████████████████████████████| 9/9 [00:00<00:00, 43.03it/s] [294/342] files, CounterServicePkg/Code/.dir Time left: 1189 seconds Uploading path: CounterServicePkg/Code/runtimes: 0it [00:00, ?it/s] [295/342] files, CounterServicePkg/Code/runtimes/.dir Time left: 1188 seconds Uploading path: CounterServicePkg/Code/runtimes/linux: 0it [00:00, ?it/s] [296/342] files, CounterServicePkg/Code/runtimes/linux/.dir Time left: 1188 seconds Uploading path: CounterServicePkg/Code/runtimes/linux/lib: 0it [00:00, ?it/s] [297/342] files, CounterServicePkg/Code/runtimes/linux/lib/.dir Time left: 1188 seconds Uploading path: CounterServicePkg/Code/runtimes/linux/lib/netstandard2.0: 100%|█| 21/21 [00:01<00:00, 17.30 [319/342] files, CounterServicePkg/Code/runtimes/linux/lib/netstandard2.0/.dir Time left: 1187 seconds Uploading path: CounterServicePkg/Code/runtimes/win: 0it [00:00, ?it/s] [320/342] files, CounterServicePkg/Code/runtimes/win/.dir Time left: 1187 seconds Uploading path: CounterServicePkg/Code/runtimes/win/lib: 0it [00:00, ?it/s] [321/342] files, CounterServicePkg/Code/runtimes/win/lib/.dir Time left: 1187 seconds Uploading path: CounterServicePkg/Code/runtimes/win/lib/netstandard2.0: 100%|█| 15/15 [00:00<00:00, 35.07it [337/342] files, CounterServicePkg/Code/runtimes/win/lib/netstandard2.0/.dir Time left: 1186 seconds Uploading path: CounterServicePkg/Data: 100%|████████████████████████████████| 1/1 [00:00<00:00, 5.69it/s] [339/342] files, CounterServicePkg/Data/.dir Time left: 1186 seconds Uploading path: CounterServicePkg/Config: 100%|██████████████████████████████| 2/2 [00:00<00:00, 18.08it/s] [342/342] files, CounterServicePkg/Config/.dir Time left: 1186 seconds Complete

[jeffj6123]

@samedder where can we check to make sure these 2 new libraries are allowed to be used?

[samedder]

@samedder where can we check to make sure these 2 new libraries are allowed to be used?

There's two options, you can merge it and see if internally it generates a bug for insecure package use. That will require a PR to follow up. Or, you can do what I did and try to find another repo already using them and verify they don't have an open issue.

I looked manually now though and it seems the packages are O.K. but I didn't check the specific version.

[samedder]

/azp run

[azure-pipelines[bot]]

Azure Pipelines successfully started running 1 pipeline(s).

[maburlik]

/azp run

[azure-pipelines[bot]]

Pull request contains merge conflicts.

[jeffj6123]

/azp run

[azure-pipelines[bot]]

Azure Pipelines successfully started running 1 pipeline(s).