search

SheffieldMLtracking / BBSRC_ohio

This is a placeholder repository for the BBSRC project; to allow us to assign tasks.
GNU General Public License v3.0
0 stars 0 forks source link

Data transfer from Ohio to Sheffield #20

Open Joe-Heffer-Shef opened 1 week ago

Joe-Heffer-Shef commented 1 week ago

We need to transfer image data from the bee trackers in the Ohio greenhouse to the University of Sheffield data storage infrastructure.

See:

Data pipeline

  1. In Ohio, the Raspberry Pis run a systemd service (timer) on a regular schedule that runs SFTP
  2. The SFTP client connects to a UOS VM (firewall exemption on port 22) (configure sshd_config to allow SFTP but dissalow SSH login)
  3. The new image data is synced to the SFTP server (lftp mirror feature)
  4. The SFTP service account on the VM has restricted permissions to write data to an ingress directory (i.e. just the home directory)
  5. a script (separate owner) running periodically, checks filetypes in the receiving folder, and then moves only the correct ones to the X: drive
  6. The data are stored on a mounted research storage volume (X:)

See these Topdesk tickets:

To do

Joe-Heffer-Shef commented 6 days ago

re: SHEF 2405 11695 @lionfish0 InfoSec ask for the Firewell exemption form to be submitted

lionfish0 commented 3 days ago

I'll go ahead and submit, but I think our chat yesterday suggests that a permanent AWS instance for handling fieldsite data transfers from Exeter, Sheffield, Ohio and Sussex, etc etc... will potentially be a better solution. Mike.

On Wed, 12 Jun 2024 at 09:10, Joe Heffer @.***> wrote:

re: SHEF 2405 11695 @lionfish0 https://github.com/lionfish0 InfoSec ask for the Firewell exemption form https://students.sheffield.ac.uk/it-services/information-security/firewall#campus to be submitted

— Reply to this email directly, view it on GitHub https://github.com/SheffieldMLtracking/BBSRC_ohio/issues/20#issuecomment-2162378980, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4MGQF7OMCMY5CIITGUHNLZG767ZAVCNFSM6AAAAABI25P6YCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSGM3TQOJYGA . You are receiving this because you were mentioned.Message ID: @.***>

--

Mike Smith, Lecturer in Probabilistic Machine Learning

Working part time (usually 8am-2pm, Monday-Thursday). Department of Computer Science University of Sheffield

lionfish0 commented 3 days ago

Joe: I'm already stuck on the first question (server name)!...

On Fri, 14 Jun 2024 at 18:19, Michael Smith @.***> wrote:

I'll go ahead and submit, but I think our chat yesterday suggests that a permanent AWS instance for handling fieldsite data transfers from Exeter, Sheffield, Ohio and Sussex, etc etc... will potentially be a better solution. Mike.

On Wed, 12 Jun 2024 at 09:10, Joe Heffer @.***> wrote:

re: SHEF 2405 11695 @lionfish0 https://github.com/lionfish0 InfoSec ask for the Firewell exemption form https://students.sheffield.ac.uk/it-services/information-security/firewall#campus to be submitted

— Reply to this email directly, view it on GitHub https://github.com/SheffieldMLtracking/BBSRC_ohio/issues/20#issuecomment-2162378980, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4MGQF7OMCMY5CIITGUHNLZG767ZAVCNFSM6AAAAABI25P6YCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSGM3TQOJYGA . You are receiving this because you were mentioned.Message ID: @.***>

--

Mike Smith, Lecturer in Probabilistic Machine Learning

Working part time (usually 8am-2pm, Monday-Thursday). Department of Computer Science University of Sheffield

--

Mike Smith, Lecturer in Probabilistic Machine Learning

Working part time (usually 8am-2pm, Monday-Thursday). Department of Computer Science University of Sheffield