search

astropy / astropy-benchmarks

Benchmarks for the astropy project
https://spacetelescope.github.io/bench/astropy-benchmarks/
BSD 3-Clause "New" or "Revised" License
7 stars 27 forks source link

Remove large io.ascii files from master branch #23

Open astrofrog opened 6 years ago

astrofrog commented 6 years ago

It would be better to generate these on-the-fly in the benchmark setup functions/methods

bsipocz commented 6 years ago

so these are the reason cloning the repo takes forever?

MSeifert04 commented 6 years ago

If they are really problematically big we might need to rewrite history to avoid keeping them as part of the clone.

astrofrog commented 6 years ago

I think the biggest part of the repo is actually the results branch - it has tens of thousands of files.

pllim commented 6 years ago

Yeah! Let's squash master into 1 commit! results branch won't be affected and should not be fetched blindly.

bsipocz commented 6 years ago

😱

Actually if it's only the small files, then cloning only master would solve the issue anyway. And while it indeed takes a while, I can very well live with the long cloning time, we don't do it in CI so local times doesn't really matter on the long run.

pllim commented 6 years ago

Results used to be in master until @astrofrog moved them to results, so that might be the reason why it is slow too? (I still don't understand the git clone magic completely.)

saimn commented 6 years ago

There is the --single-branch option which is quite useful if ones want to get only master (10s / 36Mb vs 2m 51s / 639Mb for the full clone):

~/lib 
(base) ❯ g clone --single-branch https://github.com/astropy/astropy-benchmarks.git
Cloning into 'astropy-benchmarks'...
remote: Counting objects: 480, done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 480 (delta 2), reused 3 (delta 0), pack-reused 471
Receiving objects: 100% (480/480), 36.45 MiB | 4.29 MiB/s, done.
Resolving deltas: 100% (232/232), done.

~/lib 10s
(base) ❯ du astropy-benchmarks
37M     astropy-benchmarks/.git
108M    astropy-benchmarks/benchmarks
144M    astropy-benchmarks

~/lib 
(base) ❯ rm -rf astropy-benchmarks

~/lib 
(base) ❯ g clone https://github.com/astropy/astropy-benchmarks.git
Cloning into 'astropy-benchmarks'...
remote: Counting objects: 96691, done.
remote: Compressing objects: 100% (343/343), done.
remote: Total 96691 (delta 1005), reused 1065 (delta 861), pack-reused 95473
Receiving objects: 100% (96691/96691), 639.43 MiB | 4.05 MiB/s, done.
Resolving deltas: 100% (95502/95502), done.

~/lib  2m 51s
(base) ❯ du astropy-benchmarks
643M    astropy-benchmarks/.git
108M    astropy-benchmarks/benchmarks
750M    astropy-benchmarks