search

dataverbinders / nl-open-data

A Flexible Python ETL toolkit for datawarehousing framework based on Dask, Prefect and the pydata stack
https://dkapitan.github.io/nl-open-data
MIT License
0 stars 1 forks source link

Use Supervisor to ensure long running flows #77

Closed galamit86 closed 3 years ago

galamit86 commented 3 years ago

Even with a non preemptible machine, prefect's Local Agent seems to crash sometimes.

To ensure it is always running in the background when the VM is running, Prefect recommend using Supervisor.

dkapitan commented 3 years ago

Hmmm, I think there might be something else going on which may also be the root cause of the long processing times.

On Fri, Mar 19, 2021 at 2:01 PM Amit Gal @.***> wrote:

Even with a non preemptible machine, prefect's Local Agent seems to crash sometimes.

To ensure it is always running in the background when the VM is running, Prefect recommend https://docs.prefect.io/orchestration/agents/local.html#using-with-supervisor using Supervisor http://supervisord.org/index.html.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dataverbinders/nl-open-data/issues/77, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7GYDU6727OAQJX6L3A6OTTENDMFANCNFSM4ZOYAT3Q .

galamit86 commented 3 years ago

Actually, I think it was how I started the client: SSH'ing from my terminal, then starting the client - so when my machine went to sleep, it broke the connection and the agent. I've restarted it by using SSH's command option, so it's not related to my terminal. If there are no issues in this round, then we're good. TBC.

dkapitan commented 3 years ago

In that case, perhaps give nohup a try: https://www.maketecheasier.com/nohup-and-uses/

galamit86 commented 3 years ago

So far so good. This is the local command I use to start up the remote agent:

gcloud compute ssh amitgalmail@nl-open-data-vm-1 --command '. $HOME/.ssh_init && cd nl-open-data &&  . .venv/bin/activate && nohup prefect agent local start'

This assumes that the VM is preconfigured with nl-open-data installed via poetry, and that the following file is placed at $HOME/.ssh_init:

# store this file as ".ssh_init" in the $HOME directory of the vm
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
if command -v pyenv 1>/dev/null 2>&1; then
  eval "$(pyenv init -)"
fi

export PATH="$HOME/.poetry/bin:$PATH"