[investigate] permanent links stopped working

[graphicore]

e.g. https://fontbakery.graphicore.de/browse/persistence/03757f152f79e9a8c653643974c45047b3d4c3817b1a7bee53c46a5893fb3a45:0/report.html

mentioned in #113 and needed to look into #116

SInce the dashboard is not in production, this is not a big problem. But in general, this data should persist and I'm not sure how it got lost.

[graphicore]

Ok, I just got a shell into the container and can confirm that the correct disk is mounted and that the data exists:

root@fontbakery-storage-persistence-5ffcc9cd8c-68rz9:/data/persistence_server/0375/7f15/2f79/e9a8c653643974c45047b3d4c3817b1a7bee53c46a5893fb3a45# cat meta.json 
{"reads":4,"counter":1,"instanceKeys":["0"]}

root@fontbakery-storage-persistence-5ffcc9cd8c-68rz9:/data/persistence_server/0375/7f15/2f79/e9a8c653643974c45047b3d4c3817b1a7bee53c46a5893fb3a45# cat data | head -n 2

.type.googleapis.com/fontbakery.dashboard.Files

Which is good, the question is why the storage-persistence still logs:

ERROR [GET] { NOT_FOUND: Can't find key: 03757f152f79e9a8c653643974c45047b3d4c3817b1a7bee53c46a5893fb3a45:0
    at _has.then.has (/var/javascript/node/util/StorageServers.js:406:23) name: 'NOT_FOUND', code: 5 }

I just had a quick test, the server doesn't find data after a restart, but the data is there. So there's the bug.

[graphicore]

It's improving, but now I find files packages that lose their "instanceKeys" e.g.:

in

https://fontbakery.graphicore.de/dispatcher/process/b1878c10-4b0d-4211-8831-48a45945c226

This link is failing:

https://fontbakery.graphicore.de/download/persistence/a51770aeaf3d57a835da37f38b1dde3313ad4b6d747e49255385785782f2810e:3.zip

but this one is working:

https://fontbakery.graphicore.de/download/persistence/a51770aeaf3d57a835da37f38b1dde3313ad4b6d747e49255385785782f2810e.zip

on the disc I see:

$  kubectl exec -it fontbakery-storage-persistence-66d87689df-wxkfd -- /bin/bash
# ... cd into that dir
root@fontbakery-storage-persistence-66d87689df-wxkfd:/data/persistence_server/a517/70ae/af3d/57a835da37f38b1dde3313ad4b6d747e49255385785782f2810e# cat meta.json 
{"reads":49,"counter":13,"instanceKeys":["b","c"]}

and indeed https://fontbakery.graphicore.de/download/persistence/a51770aeaf3d57a835da37f38b1dde3313ad4b6d747e49255385785782f2810e:b.zip is working.

I wonder how I dropped those other instance keys, but maybe it's just an old example and an older bug that got fixed but still manifests? Also, in some cases older keys are deleted, e.g. if the file task is reloading the files package after changing the configuration e.g. the message here "Cleaning up: deleting persistence files for key: a51770aeaf3d57a835da37f38b1dde3313ad4b6d747e49255385785782f2810e:d ".