search

Zuehlke / SHMACK

Automation and convenience in for setting up a S.H.M.A.C.K. stack with DC/OS on AWS
Apache License 2.0
18 stars 8 forks source link

Make data exchange workspace with cluster work again / improve it #16

Open mspringma opened 8 years ago

mspringma commented 8 years ago

Right now, sync'ing local files from workspace with cluster fails.

The good news: Mesosphere did improve default security setup on DC/OS 1.7 compared to prior releases. The bad news: You have to access any node in the cluster via a proxy - in particular also for SSH.

For interactive SSH sessions, that's no big deal as "dcos node ssh --master-proxy --leader" as used in ssh-into-dcos-mater.sh is already available and in use.

But for the scripts using rsync, some reworking would be needed to make them work again (infrastructure tests relying on that currently fail). It may be possible to solve this issue - but probably we should switch to a different solution altogether, e.g. using S3 - which could also help providing non-volatile storage for big inout and result data.

mspringma commented 8 years ago

Workaround for non-critical data is described in https://github.com/Zuehlke/hackzurich-sensordataanalysis/blob/master/S3ForSparkSubmit.md