search

IUSCA / sca-issues

1 stars 0 forks source link

Rtstats: /N/logs not mounted post-maintenance #32

Closed rperigo closed 4 years ago

rperigo commented 4 years ago

It appears Geode mounts have broken on the stats boxes after January maintenance. First look at logs shows a problem with the share name(s) on the geode.iu.edu FQDN (rather than the simple remount usually needed, will need to tweak config):

Jan  7 16:56:28 dev.stats.sca.iu.edu kernel: CIFS VFS: BAD_NETWORK_NAME: \\geode.iu.edu\soft
Jan  7 16:56:30 dev.stats.sca.iu.edu kernel: CIFS VFS: BAD_NETWORK_NAME: \\geode.iu.edu\logs
Jan  7 16:56:30 dev.stats.sca.iu.edu kernel: CIFS VFS: BAD_NETWORK_NAME: \\geode.iu.edu\soft
Jan  7 16:56:32 dev.stats.sca.iu.edu kernel: CIFS VFS: BAD_NETWORK_NAME: \\geode.iu.edu\logs
Jan  7 16:56:32 dev.stats.sca.iu.edu kernel: CIFS VFS: BAD_NETWORK_NAME: \\geode.iu.edu\soft
rperigo commented 4 years ago

This stuff was all moved to Geode2 this weekend. Working with Research Storage via Ticket to resolve / tweak

rperigo commented 4 years ago

See: https://kb.iu.edu/d/aoyg#linux for new official instruction. However, this does not appear to support KRB5+tickets as we've been doing. Clarifying with RS

rperigo commented 4 years ago

I went over to ask RS whether this was a simple issue with the way we're mounting the FS. Nate Lavender is under the impression it will require config change on their end to support Kerberos + Keytabs on CIFS exports, though it is unclear how in-depth the changes would need to be.

rperigo commented 4 years ago

Accidentally moved this to QA instead of the ticket directly beneath. Spoke with Stewart, and it appears there is something deeper. He originally recommended deleting the Kerberos credential caches and retrying. This did not help, but I noticed that we were unable to obtain a CIFS service ticket for either of the hostnames mentioned in the KB article. He will look into it and bring it up in an upcoming RS team meeting.

rperigo commented 4 years ago

Still awaiting word from RS. Apparently a keytab was deleted on their end, which needs work from UISO to be re-created for the service principal.

rperigo commented 4 years ago

Keytab was recreated for cifs/g2ex.rs.iu.edu. This allowed us to at least connect, however there are still some issues with the shares themselves due to the FS changes. Waiting on RS / Stewart to implement the needed changes and give me the go-ahead to test.

rperigo commented 4 years ago

Haichuan replied to the ticket asking us to try again - still no joy. Waiting to hear back on workarounds for the exports themselves not being available now that we can connect.

rperigo commented 4 years ago

We have mounts now! Rtstats-devel is back up with /N/logs and /N/soft. It should be noted that the export for /soft was changed to /software, but our mountpoints are the same on our end (e.g. /N/logs, /N/soft) for compat.