[Feature] make caching work as expected on CI

Is your feature request related to a problem? Please describe.

We didn't manage to figure out how to cache things properly in #88

Describe the solution you'd like

We fix this in a separate PR. Data from GIN should get cached in CI.

Describe alternatives you've considered

Additional context

Some avenues to explore to understand cache better (after reading these tips and tricks):

[ ] is it the lack of caching the pooch cache
[ ] is the lack of the hashes for the cell.xml files in the registry in conftest.py what is slowing the caching down?
[ ] is it the use of $HOME?

brainglobe / brainglobe-utils