Badger alternative

[zendern]

• What are you trying to do? Find a caching alternative for our current Badger usage. As part of #77 Badger issues/corruption was brought up and the fact that its probably more heavyweight than what we really need. On top of that we have other issues currently with later versions of Badger having vulnerabilities itself #54. So removing it would hopefully resolve those issues.

By removing Badger it should hopefully fix this as well #64 .

• What feature or behavior is this required for? Caching of the results from OSS Index. It is rate limited so the reason this is in place is to not overload that service and realistically deps don't change that often so caching the results for 12 hours currently helps with those things.

• How could we solve this issue? (Not knowing is okay!) Does go have a simple http request diskcache?? Serialize the contents to disk ourselves?? Find a different caching solution instead of Badger??

• Anything else?

cc @bhamail / @DarthHater

[fitzoh]

Does go have a simple http request diskcache??

• https://github.com/victorspringer/http-cache no disk based adapter (although badger is on the roadmap 😆 )
• https://github.com/gregjones/httpcache This one has a couple disk based backends:
  • github.com/gregjones/httpcache/memcache provides memcache implementations, for both App Engine and 'normal' memcache servers.
  • github.com/gregjones/httpcache/leveldbcache provides a filesystem-backed cache using leveldb.
  • github.com/birkelund/boltdbcache provides a BoltDB implementation (based on the bbolt fork).

Serialize the contents to disk ourselves??

This feels easy to mess up

Find a different caching solution instead of Badger??

• https://github.com/etcd-io/bbolt is the promariy maintained fork of https://github.com/boltdb/bolt
• sqlite might be worth considering
  • if corruption is a major concern, this is probably the most battle tested option out there
  • DELETE FROM <cache_table> makes cache cleanup easy
  • performance is relatively unimportant here

Anything else?

Maybe @DarthHater can hook us up w/ a higher rate limit if we use a Nancy user agent and forget about caching altogether.

[DarthHater]

Rate limiting is more or less here to stay, doubt we'd go that route @fitzoh (although I love the suggestion).

We should add "works on WSL (windows subsystem for linux)", etc... to our requirements, as Badger doesn't seem to work well in that context

[zendern]

64 ^^^ As @DarthHater said above .... hopefully after this is played this issue will be resolved as well.

[DarthHater]

I'm going to pick this up today. I'll see what I can figure out.

[DarthHater]

Ok, pudge is in, closing this for now!