gruntwork-io / git-xargs

git-xargs is a command-line tool (CLI) for making updates across multiple Github repositories with a single command.
https://blog.gruntwork.io/introducing-git-xargs-an-open-source-tool-to-update-multiple-github-repos-753f9f3675ec
Apache License 2.0
933 stars 63 forks source link

Improve handling of Clone Repositories into temporary folders #98

Closed plinioh closed 1 year ago

plinioh commented 1 year ago

Describe the solution you'd like

Currently, each run of git-xargs clones the desired repository into the system's temporary folder. This is true for each time you target a repository running git-xargs.

Here is an example from my machine after experiencing with git-xargs for a couple of days (MacOS 12.5.1):

Command output ```text ➜ ~ cd $TMPDIR Time: 0h:00m:00s ➜ T find . -type d -name 'git-xargs-*' -exec du -h -d 0 {} + | sort -hr 80M ./git-xargs-dgx-common-log-streaming678032190 80M ./git-xargs-dgx-common-log-streaming597087740 80M ./git-xargs-dgx-common-log-streaming3198864778 80M ./git-xargs-dgx-common-log-streaming2148875932 79M ./git-xargs-dgx-common-log-streaming3263876230 12M ./git-xargs-dgx-myaccount-identity-service1007138129 11M ./git-xargs-dgx-myaccount-identity-service3008335785 11M ./git-xargs-dgx-myaccount-identity-service2989643188 11M ./git-xargs-dgx-myaccount-identity-service2504143130 11M ./git-xargs-dgx-myaccount-identity-service2399128272 11M ./git-xargs-dgx-model-search-etl3961058151 11M ./git-xargs-dgx-model-search-etl3462385865 11M ./git-xargs-dgx-model-search-etl1808882054 11M ./git-xargs-dgx-model-search-etl1631063994 10M ./git-xargs-dgx-model-search-etl4110192775 6.4M ./git-xargs-dgx-myaccount-dashboard-service3331515564 6.4M ./git-xargs-dgx-myaccount-dashboard-service2927083683 6.4M ./git-xargs-dgx-monthlycare-service3226756721 6.2M ./git-xargs-dgx-monthlycare-service1085377643 6.0M ./git-xargs-dgx-pr-service2396586343 6.0M ./git-xargs-dgx-myaccount-dashboard-service522882412 5.7M ./git-xargs-dgx-monthlycare-service3608353224 5.7M ./git-xargs-dgx-monthlycare-service1697101609 5.7M ./git-xargs-dgx-monthlycare-service136264569 5.5M ./git-xargs-dgx-myaccount-dashboard-service762928531 5.5M ./git-xargs-dgx-myaccount-dashboard-service4145730298 5.5M ./git-xargs-dgx-myaccount-dashboard-service3958640074 5.5M ./git-xargs-dgx-myaccount-dashboard-service3647061136 5.0M ./git-xargs-dgx-pr-service538758956 5.0M ./git-xargs-dgx-pr-service461586568 5.0M ./git-xargs-dgx-pr-service3223152605 5.0M ./git-xargs-dgx-pr-service2447754502 4.8M ./git-xargs-dgx-common-services1332210037 4.6M ./git-xargs-dgx-claims-service1042501635 4.3M ./git-xargs-dgx-devops-serverless3712677518 4.3M ./git-xargs-dgx-devops-serverless1130242960 4.3M ./git-xargs-dgx-common-esb-service76139217 4.3M ./git-xargs-dgx-common-esb-service420185571 4.3M ./git-xargs-dgx-common-esb-service2481447801 4.3M ./git-xargs-dgx-common-esb-service2163038516 4.3M ./git-xargs-dgx-common-esb-service1819556725 4.2M ./git-xargs-dgx-devops-serverless492669529 4.2M ./git-xargs-dgx-devops-serverless4002798199 4.2M ./git-xargs-dgx-devops-serverless128927576 4.2M ./git-xargs-dgx-common-services491456596 4.2M ./git-xargs-dgx-common-services3582270430 4.2M ./git-xargs-dgx-common-services2000672389 4.2M ./git-xargs-dgx-common-services1293356187 4.1M ./git-xargs-dgx-claims-service51095670 4.1M ./git-xargs-dgx-claims-service1317238455 4.0M ./git-xargs-dgx-claims-service598378118 4.0M ./git-xargs-dgx-claims-service3887577741 3.6M ./git-xargs-dgx-common-basket-service2783467346 3.4M ./git-xargs-dgx-common-basket-service2102640962 3.2M ./git-xargs-dgx-randc-calendar-service608672353 3.2M ./git-xargs-dgx-common-person-service1016254311 2.8M ./git-xargs-dgx-common-basket-service649762349 2.6M ./git-xargs-dgx-randc-calendar-service1262282764 2.6M ./git-xargs-dgx-common-basket-service497612144 2.6M ./git-xargs-dgx-common-basket-service3018723509 2.5M ./git-xargs-dgx-randc-repair-record-service722209285 2.5M ./git-xargs-dgx-randc-repair-record-service52961552 2.5M ./git-xargs-dgx-randc-repair-record-service2450792822 2.5M ./git-xargs-dgx-randc-repair-record-service2170856959 2.5M ./git-xargs-dgx-randc-repair-record-service1677535228 2.5M ./git-xargs-dgx-randc-calendar-service4082317522 2.5M ./git-xargs-dgx-randc-calendar-service3869472038 2.5M ./git-xargs-dgx-randc-calendar-service2469708160 2.4M ./git-xargs-dgx-common-person-service585502068 2.4M ./git-xargs-dgx-common-person-service1949464635 2.3M ./git-xargs-dgx-common-person-service4179273804 2.3M ./git-xargs-dgx-common-person-service1961217495 2.2M ./git-xargs-dgx-randc-fault-array-service764424756 2.2M ./git-xargs-dgx-randc-fault-array-service3359491842 2.2M ./git-xargs-dgx-randc-fault-array-service335896226 2.2M ./git-xargs-dgx-randc-fault-array-service3317283419 2.2M ./git-xargs-dgx-randc-fault-array-service1527921735 2.0M ./git-xargs-dgx-claims-auth-service883203696 2.0M ./git-xargs-dgx-claims-auth-service4200032768 2.0M ./git-xargs-dgx-claims-auth-service4125151531 2.0M ./git-xargs-dgx-claims-auth-service270423822 2.0M ./git-xargs-dgx-claims-auth-service1939072047 1.9M ./git-xargs-dgx-template-serverless1388691735 1.9M ./git-xargs-dgx-template-serverless1340661728 1.8M ./git-xargs-dgx-template-serverless4110527560 1.8M ./git-xargs-dgx-template-serverless3071265855 1.8M ./git-xargs-dgx-template-serverless1845506612 1.8M ./git-xargs-dgx-skyprotect-plan-service4007101597 1.8M ./git-xargs-dgx-skyprotect-plan-service3697141120 1.8M ./git-xargs-dgx-skyprotect-plan-service3591216090 1.8M ./git-xargs-dgx-skyprotect-plan-service3342834588 1.8M ./git-xargs-dgx-skyprotect-plan-service1474255810 1.8M ./git-xargs-dgx-register-service1516419224 1.7M ./git-xargs-dgx-register-service913907664 1.7M ./git-xargs-dgx-register-service479951696 1.7M ./git-xargs-dgx-register-service2708612804 1.7M ./git-xargs-dgx-register-service2130368497 1.7M ./git-xargs-dgx-randc-model-search-service3287565133 1.7M ./git-xargs-dgx-randc-model-search-service3234429959 1.7M ./git-xargs-dgx-randc-model-search-service2381107305 1.7M ./git-xargs-dgx-randc-model-search-service1821807766 1.7M ./git-xargs-dgx-randc-model-search-service1383130522 1.7M ./git-xargs-dgx-common-test-event-service892877202 1.7M ./git-xargs-dgx-common-test-event-service674811894 1.7M ./git-xargs-dgx-common-test-event-service3637319092 1.7M ./git-xargs-dgx-common-test-event-service3577161244 1.7M ./git-xargs-dgx-common-test-event-service1721180667 1.7M ./git-xargs-dgx-common-catalogue-service983939112 1.7M ./git-xargs-dgx-common-catalogue-service2772470287 1.7M ./git-xargs-dgx-common-catalogue-service2509154114 1.7M ./git-xargs-dgx-common-catalogue-service2099321381 1.7M ./git-xargs-dgx-common-catalogue-service1891326121 1.7M ./git-xargs-dgx-common-auth-service844095786 1.7M ./git-xargs-dgx-common-auth-service834682523 1.7M ./git-xargs-dgx-common-auth-service4055385734 1.7M ./git-xargs-dgx-common-auth-service1811526230 1.7M ./git-xargs-dgx-common-auth-service1161090524 1.6M ./git-xargs-dgx-myaccount-auto-account-etl898797957 1.6M ./git-xargs-dgx-myaccount-auto-account-etl415195308 1.6M ./git-xargs-dgx-myaccount-auto-account-etl3964327510 1.6M ./git-xargs-dgx-myaccount-auto-account-etl2747648088 1.6M ./git-xargs-dgx-myaccount-auto-account-etl1831452060 1.5M ./git-xargs-dgx-model-search-service662120063 1.5M ./git-xargs-dgx-model-search-service3681460599 1.5M ./git-xargs-dgx-model-search-service3580138691 1.5M ./git-xargs-dgx-model-search-service2402385925 1.5M ./git-xargs-dgx-model-search-service1796394386 1.3M ./git-xargs-dgx-common-payment-service4043116693 1.3M ./git-xargs-dgx-common-payment-service3805125252 1.3M ./git-xargs-dgx-common-payment-service315251800 1.3M ./git-xargs-dgx-common-payment-service3049033308 1.3M ./git-xargs-dgx-common-payment-service2832457602 1.2M ./git-xargs-dgx-common-log-alert-service961588218 1.2M ./git-xargs-dgx-common-log-alert-service490772998 1.2M ./git-xargs-dgx-common-log-alert-service339914197 1.2M ./git-xargs-dgx-common-log-alert-service238092585 1.2M ./git-xargs-dgx-common-log-alert-service2337964869 1.2M ./git-xargs-dgx-common-imei-service953757373 1.2M ./git-xargs-dgx-common-imei-service537027263 1.2M ./git-xargs-dgx-common-imei-service2268986149 1.2M ./git-xargs-dgx-common-imei-service1339475111 1.2M ./git-xargs-dgx-common-imei-service1279518698 1.2M ./git-xargs-dgx-common-esb-mock-service3924556738 1.2M ./git-xargs-dgx-common-esb-mock-service367906134 1.2M ./git-xargs-dgx-common-esb-mock-service297756220 1.2M ./git-xargs-dgx-common-esb-mock-service259114694 1.2M ./git-xargs-dgx-common-esb-mock-service175502976 1.1M ./git-xargs-dgx-monthlycare-mocks3789520140 1.1M ./git-xargs-dgx-monthlycare-mocks3313344980 1.1M ./git-xargs-dgx-monthlycare-mocks2236466073 1.1M ./git-xargs-dgx-monthlycare-mocks1441079628 1.1M ./git-xargs-dgx-monthlycare-mocks1243466874 1.0M ./git-xargs-dgx-template-nx-serverless740514672 1.0M ./git-xargs-dgx-template-nx-serverless3962533842 1.0M ./git-xargs-dgx-template-nx-serverless308818190 1.0M ./git-xargs-dgx-template-nx-serverless2986665723 1.0M ./git-xargs-dgx-template-nx-serverless122325124 980K ./git-xargs-dgx-devops-pipeline-slack2687683694 980K ./git-xargs-dgx-devops-pipeline-slack2570664404 968K ./git-xargs-dgx-devops-pipeline-slack2529337648 968K ./git-xargs-dgx-devops-pipeline-slack1113530402 968K ./git-xargs-dgx-devops-pipeline-slack1065848740 964K ./git-xargs-dgx-common-log-queries3822392521 964K ./git-xargs-dgx-common-log-queries3755063549 964K ./git-xargs-dgx-common-log-queries3119748841 964K ./git-xargs-dgx-common-log-queries2477311213 964K ./git-xargs-dgx-common-log-queries2448373359 856K ./git-xargs-dgx-template-glue833851757 856K ./git-xargs-dgx-template-glue4091880147 856K ./git-xargs-dgx-template-glue3470239330 856K ./git-xargs-dgx-template-glue3382504572 856K ./git-xargs-dgx-template-glue2509518460 800K ./git-xargs-dgx-devops-deploy-workflow929715025 800K ./git-xargs-dgx-devops-deploy-workflow3833753504 800K ./git-xargs-dgx-devops-deploy-workflow2820543249 800K ./git-xargs-dgx-devops-deploy-workflow2394489380 800K ./git-xargs-dgx-devops-deploy-workflow1810561314 440K ./git-xargs-dgx-devops-cfn-certificate-provider753957369 440K ./git-xargs-dgx-devops-cfn-certificate-provider3410003968 440K ./git-xargs-dgx-devops-cfn-certificate-provider3186139486 440K ./git-xargs-dgx-devops-cfn-certificate-provider171854214 440K ./git-xargs-dgx-devops-cfn-certificate-provider1533139722 ```

As one can check, there are multiple instances of the same repository, which can lead to extreme sizes after using git-xargs for a long time.

Describe alternatives you've considered

I see a couple of alternatives to resolve this issue:

  1. Clean up the repos after each execution from git-xargs. Add a optional flag --keep-cloned-repositories to keep the repo for debugging purposes.

  2. Make the CLI check if the repositories that need cloning are not already present in the filesystem. If they are, perform a git pull instead of cloning the repository again. This one will probably need changing quite a lot in the code, but can lead to improvements on performance, specially when dealing with large repositories.

Additional context

N/A

zackproser commented 1 year ago

Great bug report, thank you!

I think both of your suggestions are reasonable.

zackproser commented 1 year ago

@plinioh , I put up a pull request (linked above) to address this - if you have a free moment please give it a shot!

Thanks

pliniodng commented 1 year ago

Looks good, looking forward to the next release (:

Thanks @zackproser