woodRock / psychic-invention

NZODN Data Ingestion Project
0 stars 0 forks source link

Test SFTP Server #25

Closed woodRock closed 3 years ago

woodRock commented 3 years ago

Goal

Test access the to the Niwa SFTP server.

Tasklist

Success Criteria

We have remote access to the SFTP server. And sufficient permissions to perform the jobs needed for the ingestion.

woodRock commented 3 years ago
email
woodRock commented 3 years ago

Background

This space on the SFTP server is where the CTD data will be pushed to. We must write a CRON job on the wellimos server. This job pulls that data from the SFTP server, ingests it into PostgreSQL, then deletes it from the SFTP server if successful.

SSH Keys

We all sent out public SSH keys to IT. There is a generic service account that can be accessed using any of our SSH keys. Niwa decided to migrate to Secure File Transfer Protocol (SFTP), rather than FTP, for security reasons.

Test Directory

The test directory is for us to play around with. When testing initial cron jobs, and workflows for data ingestion.

CRUD Functionality

We tested that the service account role has the sufficient permissions that will be required for the ingestion. These details will be passed using a SSH agent for the bash scripts that perform the ingestion. For security reasons we wish to avoid storing the private keys for any of our users as plain text.

woodRock commented 3 years ago

We have performed all the relevant tests to ensure that the access to the SFTP server is not an issue.