Incorrect behavior when pinned SFTP key changes on running instance

[jrnt30]

Had a bit of a weird one recently with v0.23.4. We have explicitly pinned our valid SSH key for an SFTP host in our configuration. During the course of the application being up, our banking partner adjusted their public key that we had previously pinned to a different value and the application continued to operate as if things were OK.

Observed Behavior

• The running instance of ACH Gateway did not have any logs indicating failure for key validation (upon restarting the application, we did see those we would expect indicating failure)
• The merged files were considered "successfully" processed and uploaded which resulted in the merged files cleaned up, the Kafka message around file delivery being created and the files themselves being uploaded properly to the "audit" directory
• There were no files uploaded nor downloaded from the actual SFTP server

Expected Behavior

• Error messages would be triggered upon failure to upload or download (similar to what happens when we restarted the application)
• The application would not delete the merged files locally upon failure to actually upload them
• No kafka messages would be emitted
• Audit files would not be uploaded

[adamdecaf]

That's quite odd. I'm assuming the connection was left open between the key rotation? I don't understand why the SFTP server would not save files, but also wouldn't tell you of such errors. I'm not sure how the SFTP library could detect a changed key or that files weren't actually written (without trying to open each file after write).

[adamdecaf]

HostKeyCallback is only called by the ssh client during the handshake, so if there's no further handshakes the client wouldn't notice that the key changed.

We could be more aggressive about closing upload.Agent's but that comes at a performance cost. I've looked at closing upload.Agent's and removing the cache which would help detect this sooner.