search

moov-io / paygate

A RESTful API enabling electronic payments to be submitted and received without a deep understanding payment file specification
http://moov.io
Apache License 2.0
130 stars 31 forks source link

build: add FTP and SFTP servers in docker-compose #161

Open adamdecaf opened 5 years ago

adamdecaf commented 5 years ago

We should launch our docker-compose setup with an FTP and SFTP (ssh file transfer protocol) servers running so paygate can show the async file transfer logic. Instead paygate currently spams the log with errors:

{"caller":"file_transfer_async.go:215","startPeriodicFileOperations":"Starting periodic file operations","ts":"2019-07-06T00:47:05.870453375Z"}
...
{"caller":"file_transfer_async.go:271","downloadAndProcessIncomingFiles":"error downloading files into /storage/downloaded469047369","error":"downloadAllFiles: problem with 323075822 file transfer agent init: EOF","ts":"2019-07-06T00:49:05.879605236Z"}

There are commands in the makefile, make start-ftp-server and make start-sftp-server we can use that as a starting point for the docker-compose file. The only complication is that inbound and returned files are deleted, so you’d only see paygate process files in the first iteration of its sftp loop.

This also ties in with https://github.com/moov-io/paygate/issues/147 as to how we would configure this inside the docker-compose file.

wadearnold commented 5 years ago

Should this be a part of a larger... "run as a developer"?

For production, you would want to know that an SFTP endpoint was properly configured. The error that I see in the log is what I would want to have for production. I also think that a "system check" script would be useful for production to ensure that the system was properly configured.

adamdecaf commented 5 years ago

Maybe the issue wasn't clear, but paygate defaults to checking the FTP/SFTP server from make start-[s]ftp-server -- even if that server isn't running. See: https://github.com/moov-io/paygate/blob/533fdf8bec6516ab3cd496ea1342673d80f7de5c/file_transfer_configs.go#L174-L183

The error logs would still appear on connection errors in production. We don't log differently for production vs development.

We could add each server to :9092/live, which Kubernetes uses to monitor paygate's health.

$ curl -s localhost:9092/live | jq . 
{
  "accounts": "accounts ping failed: Get http://localhost:8085/ping: dial tcp [::1]:8085: connect: connection refused",
  "ach": "error getting /ping from ACH service: ACH GET requestId=50d190562c0e1b8423c2d0b521052984fad41c72 : Get http://localhost:8080/ping: dial tcp [::1]:8080: connect: connection refused",
  "fed": "FED ping failed: Get http://localhost:8086/ping: dial tcp [::1]:8086: connect: connection refused",
  "ofac": "ofac.Ping: failed: Get http://localhost:8084/ping: dial tcp [::1]:8084: connect: connection refused"
}

Adding fields like "sftp-localhost:2121": null (or an error string).