Allow git + hex deps to be cached

[artman41]

• Enabled via CACHE_DEPS=1
• Cached files are beneath $(HOME)/.cache/erlang.mk/ by default
• Will cache Git Fetches (including submodules) & Hex fetches
  • These cache in $(CACHE_DIR)/git & $(CACHE_DIR)/hex

[artman41]

The reasoning behind this MR is that, recently, I've been doing a lot of offline dev & it's quite frustrating when you know you have the git repo in a project somewhere locally, but not sure where.

Aside from offline benefits, this MR should also speed up subsequent deps calls after a distclean if you're on a slow connection & using the cache.

[essen]

Not a bad idea but I think you are missing a git pull when the repository already exists otherwise it might not be able to checkout new branches/tags?

[artman41]

Not a bad idea but I think you are missing a git pull when the repository already exists otherwise it might not be able to checkout new branches/tags?

Good shout, threw a git pull in there after the git fetch - anything else you can think of?

Unsure whether it's worthwhile symlinking from the $(CACHE_DIR) or whether using it as a clone source is good enough

[essen]

A clone would work out better across projects, for example if one project depends on a version and another a different version. Which means that the git checkout after the pull is probably not necessary. Also git pull includes git fetch so that's not necessary either, one or the other should be good enough. Not sure if you had git fetch before, maybe I didn't look hard enough.

Other than that, I think having some tests would be good. I'll see if some folks want to try it tomorrow.

[artman41]

A clone would work out better across projects, for example if one project depends on a version and another a different version. Which means that the git checkout after the pull is probably not necessary. Also git pull includes git fetch so that's not necessary either, one or the other should be good enough. Not sure if you had git fetch before, maybe I didn't look hard enough.

Other than that, I think having some tests would be good. I'll see if some folks want to try it tomorrow.

Sounds good, I've had it in the past where a pull only pulled the latest refs for the branches I had locally (though I guess if we change it to git pull --all then it'll likely pull everything)

• I Agree with the cloning over symlinking.

Ref the git checkout - you're probably right that we can remove that, now we're using it as a local git repo, leaving it checked out on to a branch doesn't really have a purpose for us.

What're you thinking test wise?

My current idea is the following (otherwise I'm not sure how you'd be able to tell you've cloned it from the cache only)

• Check dep doesn't exist in cache
• Clone dep in
• Check dep does exist in cache
• Make a commit to the cache version only
• distclean
• Clone deps in again, check cache commit is there

I also had a realisation while typing this that it'd likely be worthwhile to have a target to clear the cache (maybe with the ability to specify a specific dep?) - what're your thoughts on that?

[essen]

The tests have code you can reuse that creates repos. I would not recommend modifying the cache within the test since we want to test the cache as well not just the code. There's probably a bunch of different scenarios to try, but separate git/subfolder/hex would be a good start. Also testing commits, branches and tags and making sure different projects can depend on different versions and they all can use the same cache.

About clearing the cache, nothing beats rm -rf ~/.cache/erlang.mk/...

[artman41]

The tests have code you can reuse that creates repos. I would not recommend modifying the cache within the test since we want to test the cache as well not just the code. There's probably a bunch of different scenarios to try, but separate git/subfolder/hex would be a good start. Also testing commits, branches and tags and making sure different projects can depend on different versions and they all can use the same cache.

About clearing the cache, nothing beats rm -rf ~/.cache/erlang.mk/...

• New target cacheclean
• 2 Tests, 1 to test that the cache works & 1 to test that 2 apps can reuse the cache for different versions of the same dep

[essen]

Merged with some tweaks and a fix for Hex which couldn't fetch if the package was in the cache (misplaced ;). Thanks!