openedx / openedx-atlas

An Open edX CLI tool for moving translation files from openedx-translations
GNU Affero General Public License v3.0
3 stars 7 forks source link

perf: 120x faster clone with real-world performance tests #14

Closed OmarIthawi closed 1 year ago

OmarIthawi commented 1 year ago

Description

git add remote --fetch origin is very slow for any medium to large repository. With this fix, the performance is hugely improved by fetching only after sparse-checkout rules are set, therefore making checkout almost instant.

Additionally, the git steps are simplified into fewer steps.

Testing

Test results

The new_ implementation is waaaay faster!

Test Old New
Clone edx-ios-app Source directory 1m 57s 1s
Clone edx-platform Too much 2s
Performance testing code

```shell new_clone_ios() { cd /tmp rm -rf /tmp/atlas-repo mkdir -p /tmp/atlas-repo cd /tmp/atlas-repo git clone --filter=blob:none --no-checkout --depth=1 --branch=master "https://github.com/openedx/edx-app-ios.git" atlas-repo cd atlas-repo git sparse-checkout set '!*' git sparse-checkout add 'Source/**' git checkout HEAD tree } old_clone_ios() { cd /tmp rm -rf /tmp/atlas-repo mkdir -p /tmp/atlas-repo cd /tmp/atlas-repo git init -b master git remote add -f origin "https://github.com/openedx/edx-app-ios.git" git sparse-checkout set '!*' git sparse-checkout add 'Source/**' git pull origin master tree } new_clone_edx_platform() { cd /tmp rm -rf /tmp/atlas-repo mkdir -p /tmp/atlas-repo cd /tmp/atlas-repo git clone --filter=blob:none --no-checkout --depth=1 --branch=master "https://github.com/openedx/edx-platform.git" atlas-repo cd atlas-repo git sparse-checkout set '!*' git sparse-checkout add 'conf/locale/ar**' git checkout HEAD tree } old_clone_edx_platform() { cd /tmp rm -rf /tmp/atlas-repo mkdir -p /tmp/atlas-repo cd /tmp/atlas-repo git init -b master git remote add -f origin "https://github.com/openedx/edx-platform.git" git sparse-checkout set '!*' git sparse-checkout add 'conf/locale/ar**' git pull origin master tree } ################## time new_clone_ios # real 0m1.147s # user 0m0.064s # sys 0m0.016s ################## time old_clone_ios # real 1m57.448s # user 0m35.059s # sys 0m9.742s ################### time new_clone_edx_platform # real 0m2.600s # user 0m0.237s # sys 0m0.079s ################## time old_clone_edx_platform # real infinity! # user infinity! # sys infinity! ```

References

This pull request is part of the FC-0012 project which is sparked by the Translation Infrastructure update OEP-58.

Check the links above for full information about the overall project.

Internalization is being rearchitected in Open edX Python, XBlock, Micro-frontend, and other projects. There are a number of immediately visible changes:

Breaking Changes

One of the primary goals of the project is to avoid breaking changes. If you noticed any suspicious code, please raise your concern. But before that, please know the strategy we're following to avoid breaking changes.

openedx-webhooks commented 1 year ago

Thanks for the pull request, @OmarIthawi! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

OmarIthawi commented 1 year ago

@brian-smith-tcril this is now ready for review.

openedx-webhooks commented 1 year ago

@OmarIthawi 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.