search

nstudach / sugar

Automatic set up of DigitalOcean VM's and execution of pathspider with upload to MAMI PTO
0 stars 0 forks source link

Feature: Hellfire integration #8

Closed nstudach closed 6 years ago

nstudach commented 6 years ago

Integrate hellfire for automatic host list download and splitting via input_modifier.py

nstudach commented 6 years ago

Tests have shown that uploading it from a server or locally to each measurement-server is unfeasible for large input files. It would be necessary to integrate an automatic upload to a cloud storage such as dropbox.

buehlert commented 6 years ago

In case you keep the two branches (not a problem at all), make sure that the readme files clearly describe the differences.

nstudach commented 6 years ago

The hellfire branch was merged into master this morning. The speed of the copy function was increased by compressing the data first (~10x smaller) and using scp to copy instead of the ParallelSSH copy function (way faster).