Matrix builds not working

[phlummox]

Whatever software is behind https://matrix.hackage.haskell.org currently seems not to be working - I've tested from my office desktop machine browser, from a cloud VM I have access to in Singapore, and using https://httpstatus.io.

For example: looking at the 'vector' package under the 'Status' heading on the right-hand side, I should be able to get to a matrix of build results for that package by following the link to https://matrix.hackage.haskell.org/package/vector; but instead, the bulk of that page seems to be just an empty div.

The URL https://matrix.hackage.haskell.org/api/v2/packages/vector/badge should serve up a status "badge" for vector as an SVG, but instead, that URL currently gives a 500 status error, plus the error text:

A web handler threw an exception. Details:
libpq: failed (FATAL:  could not open file "base/16582/2601": Read-only file system
)

From looking at the Wayback Machine, it seems that the badge URL was working on 18th December --

curl http://web.archive.org/web/20211218201456/https://matrix.hackage.haskell.org/api/v2/packages/vector/badge

correctly gives back an SVG image.

I also note that http://status.haskell.org/ currently reports that hackage is "Operational" - which I guess is true, but perhaps it might be worth adding a check of matrix.hackage.haskell.org to that health check as well?

[phlummox]

Oh - I just realized that http://status.haskell.org/ says "page is manually updated with information regarding service status by haskell.org admins", which doesn't seem to be an ideal way to do things, but okay.

However the link that page provided to what I presume is an automated health-check service, https://auto-status.haskell.org seems to be reporting a 404 error. Is the link outdated? Does it need to be changed?

I'm not sure if haskell.org problems like this should be reported somewhere other than here - let me know if so.

[phlummox]

From what I can tell, it seems that https://github.com/haskell-infra/haskell-admins is the place to report problems with http://status.haskell.org/, so I've raised an issue over there: https://github.com/haskell-infra/haskell-admins/issues/9

[gbaz]

cc @andreasabel (who has been looking to take over some matrix maintenance from herbert, who has been too busy to deal with it)

Fixing the auto-status url is a longtime admin todo I'm planning to get to soon (we have to manage the custom dns ourselves because the service we used moved that to a paid only plan)

[phlummox]

Cheers. I have sent an email to admin@haskell.org inquiring about helping out - not sure how much time might be required, but thought I'd check.

[andreasabel]

@gbaz it is more than six weeks ago that I got hold of Herbert last time; I reported that matrix.hackage is down but get no answer.

[gbaz]

update: one of the disks is borked. davean thinks he can restore from backup, but isn't sure about timing.

[Bodigrim]

CC @myShoggoth

[gbaz]

candidate replacement (not an improvement, just getting everything working again on a new box) at https://matrix.haskell.org -- we plan to swap it over soon, if anyone wants to poke it a bit first and see if they're comfortable with it.

[sjakobi]

@gbaz It seems that recent releases and revisions are not built and don't show up in the result matrices.

For example https://hackage.haskell.org/package/mandrill-0.5.6.0 does not show up in https://matrix.haskell.org/#/package/mandrill.

https://matrix.haskell.org/#/latest also looks badly out of date – the latest builds are from December.

[re-xyr]

Hi, are there any updates with this outage? Is there anything blocking this from a fix?

[Bodigrim]

@gbaz any chance to reenable matrix builds soon?

[gbaz]

No updates to give -- the migrated box doesn't work right, and nobody has had time to investigate why. We need a good sysadmin volunteer to step up and work on this (or to find an entity willing to assign one).