persistence git clone storage

[rucciva]

Hi, first of all thanks a lot for creating this.

one thing i noticed is that the cloned git repository is stored in-memory. If we were to run it as standalone server, then as long as the server doesn't exit, the cloned state still persist, meanwhile if using the cli wrapper, it needs to git clone every time it runs.

the problem is, it doesn't take too long for the git repo to grow in size and usually means slow download time. If we were to use cli wrapper, then the execution time will grow.

so is it possible to store the git clone persistently? or is there any workaround?

---

i did try using LFS but it seems to change the state and make it undecryptable (i'm currently using AES enryption)

[dee-kryvenko]

Hi @rucciva - thanks for interest in this project and for the feedback. You are right - this will become a pain on big repositories with long history, I haven't thought of that. The intention was to avoid dealing with cleanup if I were to use real filesystem. Should be easy enough to change that behavior - just a matter of giving it different fs implementation in https://github.com/plumber-cd/terraform-backend-git/blob/v0.1.5/storages/git/git.go#L160-L172. I will try to look into it when I get some free time.

[rucciva]

Hi @dee-kryvenko , thanks for the kind respond. Looking forward to it. Regarding the cleanup, might i suggest adding flag or env that signal the executable to delete the cloned directory as cleanup before doing the main job?

[rucciva]

Hi @dee-kryvenko , i've been thinking, what if instead of commiting to $ref each time writing state, the backend do the following:

1. Copy the existing state to state-$unixtimestamp
2. Delete old state- file so that only N latest state- file remained
3. Write the new state
4. Git commit --amend
5. Git force push

This way, we don't keep appending to git history of the state which in most case is no longer used (unlike the history of the terraform code it self) and save storage.

If you approve the design, i would like to try to contribute

[toabi]

Oh. This behaviour explains why ours gets so slow and sluggish and eats 2.5Gi RAM…

[dee-kryvenko]

Hi @dee-kryvenko , i've been thinking, what if instead of commiting to $ref each time writing state, the backend do the following:

1. Copy the existing state to state-$unixtimestamp
2. Delete old state- file so that only N latest state- file remained
3. Write the new state
4. Git commit --amend
5. Git force push

This way, we don't keep appending to git history of the state which in most case is no longer used (unlike the history of the terraform code it self) and save storage.

If you approve the design, i would like to try to contribute

Hi @rucciva, sorry - somehow I missed your last message with this proposal. I just saw your PR that implements commit with amend option and a force push. It should theoretically work due to atomic locking, but I always get shivers when someone mentioning force push. Besides, the history which is a native function of git - would be lost. You mentioned initially the option to persist the state locally instead of in-memory clones. Did you had a chance to test if it helps much with the original issue? If yes, it would be trivial to make the backend use FS instead of memory thus avoiding full clones and relying more on fetches, albeit some extra legwork will be needed for cleanups but IMO is a better problem to have than a force push.

Alternatively, what I was thinking is GitHub (and most alternatives) usually comes with HTTP API that allows basic CRUD operations. Which in the nutshell makes it an object storage akin to S3, although with much lower rate limits. It should be possible to use HTTP API for the purposes of this backend instead of Git protocol. Sadly it would mean separate implementation per each Git provider as they all have different API. Which one are you using btw?

Lastly, if we do go with amend and force push route, as it would take away from existing functionality (ability to trace state changes over time via git history) and in general would be a breaking change, I suggest this should be a new toggleable option disabled by default.

Thoughts?

[toabi]

Hm… if it would force push like the commandline with --force-with-lease it would be reasonably safe I guess in addition to the other conflict checking done by terraform itself?

[dee-kryvenko]

I'd have to check if this is something go-git supports, it will make it a bit safer but it still does not address the point that there will be no history of how the state was changing over time in the repository. It is not necessarily important to everyone, just like with having versioning on S3 on or off - some might have it and some might not care, so I feel like there should be an option to enable or disable this behavior. And at least try to see if we can get original issue fixed in any other way avoiding force push of any kind altogether.

[rucciva]

hi @dee-kryvenko thanks for taking your time reviewing my PR

Did you had a chance to test if it helps much with the original issue?

not yet. I realized that the persistence mechanism is just a workaround for the actual problem, which is git repository storage growth. so i abandoned the idea

Besides, the history which is a native function of git - would be lost

i was thinking that this wasn't a concern previously since other backend might not also provide versioning on the state. But if it is, then we could tag the latest cloned commit and push the tag before doing git amend and force push. The tag could be a combination of the branch name, state name, and the unix time. And we could control how many tag we keep by deleting older tag.

Sadly it would mean separate implementation per each Git provider as they all have different API. Which one are you using btw?

i am currently using self-hosted gitlab

Lastly, if we do go with amend and force push route, as it would take away from existing functionality (ability to trace state changes over time via git history) and in general would be a breaking change, I suggest this should be a new toggleable option disabled by default.

yeah i was thinking about this too, but hesitate on where to put the setting, will it be inside the url or environment variable. what do you think?