[bug] Investigate LFS (Large File Storage) usage within cncf/devstats and cncf/gitdm

[jeefy]

Currently, Devstats consumes quite a lot of GitHub's LFS offering (both in storage and in bandwidth). This doesn't seem right and should be looked into and, if possible, remedied.

cc @caniszczyk

[lukaszgryglicki]

This was approved a while ago and is only ebcause we are storing full history for JSON file changes, I can ytry to find a way to delete history for large files and only keep the most recent up to date version, I have no other ideas of how to deal with it.

[caniszczyk]

I think keeping the last few recent versions is probably fine, you wouldn't need more for debugging right?

On Tue, Mar 28, 2023 at 9:27 AM Łukasz Gryglicki @.***> wrote:

This was approved a while ago and is only ebcause we are storing full history for JSON file changes, I can ytry to find a way to delete history for large files and only keep the most recent up to date version, I have no other ideas of how to deal with it.

— Reply to this email directly, view it on GitHub https://github.com/cncf/devstats/issues/397#issuecomment-1487001040, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAPSILWOCNZZEKXYCGA343W6LYOBANCNFSM6AAAAAAWKUXDIY . You are receiving this because you were mentioned.Message ID: @.***>

-- Cheers,

Chris Aniszczyk https://aniszczyk.org

[lukaszgryglicki]

Yeah, but I think we should keep some history, say up to 1 month, but I'm not sure how to do this - this requires rewriting history erach time I commit, not sure how to do this, will investigate on Friday or later. Is this OK?

[caniszczyk]

No rush to investigate - one idea is using another repo for history data?

On Tue, Mar 28, 2023 at 9:31 AM Łukasz Gryglicki @.***> wrote:

Yeah, but I think we should keep some history, say up to 1 month, but I'm not sure how to do this - this requires rewriting history erach time I commit, not sure how to do this, will investigate on Friday or later. Is this OK?

— Reply to this email directly, view it on GitHub https://github.com/cncf/devstats/issues/397#issuecomment-1487007126, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAPSIL7NLQQMM2Y2XQUGLDW6LY3ZANCNFSM6AAAAAAWKUXDIY . You are receiving this because you were mentioned.Message ID: @.***>

-- Cheers,

Chris Aniszczyk https://aniszczyk.org

[lukaszgryglicki]

Yeah, I can make an archive of this (without LFS) and start a fresh one, it will fill up to this level in another 2 or 3 years only. makes sense? It will also make clonign a lot faster.

[caniszczyk]

that would be awesome, I think this is a better design tbh

On Tue, Mar 28, 2023 at 9:52 AM Łukasz Gryglicki @.***> wrote:

Yeah, I can make an archive of this (without LFS) and start a fresh one, it will fill up to this level in another 2 or 3 years only. makes sense? It will also make clonign a lot faster.

— Reply to this email directly, view it on GitHub https://github.com/cncf/devstats/issues/397#issuecomment-1487043194, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAPSIOOXYUA53HWJBWIMXLW6L3LJANCNFSM6AAAAAAWKUXDIY . You are receiving this because you were mentioned.Message ID: @.***>

-- Cheers,

Chris Aniszczyk https://aniszczyk.org

[lukaszgryglicki]

So this will be simple, if I have acess to create/rename repos on cncf org - pls make sure I have and then I can do it on Friday.

[jeefy]

You have full admin access to the CNCF Org 🎉

[lukaszgryglicki]

• I have renamed devstats to devstats.archive and created a fresh new devstats repo with just the newest JSON.
• I have renamed gitdm to gitdm.archive and created a fresh new gitdm repo with just the newest JSONs.

I will now archive old ones, new ones are not using LFS right now (I will try to avoid) and I will be trying to update JSONs less frequently to save space & transfer, but we still need to process affiliations so this cannot be avoided.

[lukaszgryglicki]

I'm closing this issue and archiving this repository.