Schedule ETL Tasks using Meltano

[rajasekhar-gundala]

Hell Team, Thank you very much for the project. is there any way to integrate Meltano ETL Pipelines with this tool?

[yohamta]

Hello @rajaseg, it's great to hear that you're interested in integrating Meltano ETL Pipelines with Dagu!

I'm not very familiar with Meltano, so could you please let me know what exactly you're trying to achieve?

If you're looking to run Meltano commands using Dagu, you can easily do so by creating a DAG to call the Meltano commands.

Here's an example:

schedule: "0 0 * * *" # run at 00:00 every day
steps:
  - name: myStep
    command: meltano elt <extractor_name> <loader_name> --transform <transform_name>

You can add more steps and specify dependencies of those steps, and it is visualized and managed on the Dagu's web UI.

I hope this helps! Let me know if you have any further questions or if there's anything else we can do to assist you.

[rajasekhar-gundala]

@yohamta I got below error when I try to run Dagu using docker-compose

! dagu The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v4) and no specific platform was requested

docker-compose.yaml

version: "3.7"

services:
  dagu:
    image: yohamta/dagu:latest
    restart: unless-stopped
    ports:
      - "8000:8080"
    networks:
      - caddy
    volumes:
      - ./dagu-dags:/home/dagu/.dagu/dags
      - ./dagu-data:/home/dagu/.dagu/data
      - ./dagu-logs:/home/dagu/.dagu/logs
volumes:
  dagudags:
  dagudata:
  dagulogs:
networks:
  caddy:
    external: true

After some research I have included platform: linux/arm64 entry in docker-compose I got below error after the platform entry

exec /bin/sh: exec format error

Am I doing anything wrong?

[yohamta]

Hey @rajaseg ! I'd recommend giving the following Docker image a try: yohamta/dagu:1.10.4-amd64 or yohamta/dagu:latest-amd64. It should work well for your system.

[rajasekhar-gundala]

Hey @rajaseg ! I'd recommend giving the following Docker image a try: yohamta/dagu:1.10.4-amd64 or yohamta/dagu:latest-amd64. It should work well for your system.

Hi @yohamta, when I use yohamta/dagu:latest-amd64 got the below error message

Error response from daemon: manifest for yohamta/dagu:latest-amd64 not found: manifest unknown: manifest unknown

When I use yohamta/dagu:1.10.4-amd64 got below error message

Error response from daemon: image with reference yohamta/dagu:1.10.4-amd64 was found but does not match the specified platform: wanted linux/amd64, actual: linux/arm64/v8

[yohamta]

Hi @rajaseg , thanks for providing all the details.

I've gone ahead and updated the image to make sure it's all set. Could you please give it another shot? Also, kindly double-check that your docker-compose file doesn't have platform: linux/arm64 in it.

Here are the commands to run:

docker image rm yohamta/dagu:latest-amd64
docker image rm yohamta/dagu:1.10.4-amd64
docker-compose up

[rajasekhar-gundala]

Hi @rajaseg , thanks for providing all the details.

Could you please give it another shot? Also, kindly double-check that your docker-compose file doesn't have platform: linux/arm64 in it.

Here are the commands to run:

docker image rm yohamta/dagu:latest-amd64
docker image rm yohamta/dagu:1.10.4-amd64
docker-compose up

Hi @yohamta No luck after removing the images.

I confirm that there is no platform: linux/arm64 entry in the docker-compose file. I used docker system prune -a to remove unused components in the server. Please find the screen shots below.

docker-compose

version: "3.7"

services:
  dagu:
    image: yohamta/dagu:latest-amd64 or yohamta/dagu:1.10.4-amd64
    restart: unless-stopped
    ports:
      - "8000:8080"
    networks:
      - caddy
    volumes:
      - ./dagu-dags:/home/dagu/.dagu/dags
      - ./dagu-data:/home/dagu/.dagu/data
      - ./dagu-logs:/home/dagu/.dagu/logs
volumes:
  dagu-dags:
  dagu-data:
  dagu-logs:
networks:
  caddy:
    external: true

[yohamta]

Hi @rajaseg , that's a bit strange!

Would you mind checking if any of the binaries from the link below work on your PC?

https://github.com/yohamta/dagu/releases/tag/v1.10.4

Just download the appropriate binary and give it a try with this command: ./dagu server --port=8000

Let me know how it goes!

[rajasekhar-gundala]

Hi @rajaseg , that's a bit strange!

Would you mind checking if any of the binaries from the link below work on your PC?

https://github.com/yohamta/dagu/releases/tag/v1.10.4

Just download the appropriate binary and give it a try with this command: ./dagu server --port=8000

Let me know how it goes!

Hi @yohamta Please find the outcome below.

[yohamta]

Hey @rajaseg, great to hear that it's working on your computer! Could you please let us know which binary you used?

• dagu_1.10.4_Darwin_arm64.tar.gz
• dagu_1.10.4_Darwin_x86_64.tar.gz
• dagu_1.10.4_Linux_arm64.tar.gz
• dagu_1.10.4_Linux_armv7.tar.gz
• dagu_1.10.4_Linux_x86_64.tar.gz

I can build a docker image for that binary.

[rajasekhar-gundala]

Hey @rajaseg, great to hear that it's working on your computer! Could you please let us know which binary you used?

• dagu_1.10.4_Darwin_arm64.tar.gz
• dagu_1.10.4_Darwin_x86_64.tar.gz
• dagu_1.10.4_Linux_arm64.tar.gz
• dagu_1.10.4_Linux_armv7.tar.gz
• dagu_1.10.4_Linux_x86_64.tar.gz

I can build a docker image for that binary.

Hi @yohamta I have used [dagu_1.10.4_Linux_x86_64.tar.gz](https://github.com/dagu-labs/dagu/releases/download/v1.10.4/dagu_1.10.4_Linux_x86_64.tar.gz)

[yohamta]

Hey @rajaseg, I appreciate the extra info you provided! I've figured out the issue and have rebuilt the most recent image.

Could you kindly give yohamta/dagu:latest a try instead of yohamta/dagu:latest-amd64 ? Here are the steps:

docker image rm yohamta/dagu:latest
docker image rm yohamta/dagu:1.10.4
docker-compose up

I believe it should work now! Take care and have a great day!

[rajasekhar-gundala]

Hey @rajaseg, I appreciate the extra info you provided! I've figured out the issue and have rebuilt the most recent image.

Could you kindly give yohamta/dagu:latest a try instead of yohamta/dagu:latest-amd64 ? Here are the steps:

docker image rm yohamta/dagu:latest
docker image rm yohamta/dagu:1.10.4
docker-compose up

I believe it should work now! Take care and have a great day!

Hi @yohamta I tried with yohamta/dagu:latest but no luck. docker logs dagu gives me the below error.

exec /bin/sh: exec format error

[yohamta]

Hey @rajaseg, would you mind providing me with the output of the following command?

docker image inspect yohamta/dagu:latest | grep -A2 Architecture

Also, can you please try using the platform: linux/amd64 parameter? Here's the updated docker-compose file with the platform: linux/amd64 parameter added:

version: "3.7"

services:
  dagu:
    platform: linux/amd64
    image: yohamta/dagu:latest
    restart: unless-stopped
    ports:
      - "8000:8080"
    networks:
      - caddy
    volumes:
      - ./dagu-dags:/home/dagu/.dagu/dags
      - ./dagu-data:/home/dagu/.dagu/data
      - ./dagu-logs:/home/dagu/.dagu/logs
volumes:
  dagudags:
  dagudata:
  dagulogs:
networks:
  caddy:
    external: true

Please make sure to remove the image before running docker-compose with these changes:

docker image rm yohamta/dagu:latest
docker image rm yohamta/dagu:1.10.4
docker-compose up

Thanks a lot for your help!

[rajasekhar-gundala]

Hey @rajaseg, would you mind providing me with the output of the following command?

docker image inspect yohamta/dagu:latest | grep -A2 Architecture

Also, can you please try using the platform: linux/amd64 parameter? Here's the updated docker-compose file with the platform: linux/amd64 parameter added:

version: "3.7"

services:
  dagu:
    platform: linux/amd64
    image: yohamta/dagu:latest
    restart: unless-stopped
    ports:
      - "8000:8080"
    networks:
      - caddy
    volumes:
      - ./dagu-dags:/home/dagu/.dagu/dags
      - ./dagu-data:/home/dagu/.dagu/data
      - ./dagu-logs:/home/dagu/.dagu/logs
volumes:
  dagudags:
  dagudata:
  dagulogs:
networks:
  caddy:
    external: true

Please make sure to remove the image before running docker-compose with these changes:

docker image rm yohamta/dagu:latest
docker image rm yohamta/dagu:1.10.4
docker-compose up

Thanks a lot for your help!

Hi @yohamta I also tried with platform: linux/amd64 option. But the same issue. Please find the screen shot below.

[yohamta]

Hi @rajaseg,

I'm sorry to hear that the image isn't working on your computer. Would you mind building the Docker image on your end with the following Dockerfile? It should help us determine the issue and get it resolved.

Dockerfile example:

FROM --platform=$BUILDPLATFORM alpine:latest

ARG TARGETARCH
ARG VERSION=1.3.15 
ARG RELEASES_URL="https://github.com/yohamta/dagu/releases"

ARG USER="dagu"
ARG USER_UID=1000
ARG USER_GID=$USER_UID

EXPOSE 8080

RUN <<EOF
    #User and permissions setup
    apk update
    apk add --no-cache sudo tzdata
    addgroup -g ${USER_GID} ${USER}
    adduser ${USER} -h /home/${USER} -u ${USER_UID} -G ${USER} -D -s /bin/ash
    echo ${USER} ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/${USER}
    chmod 0440 /etc/sudoers.d/${USER}
EOF

USER dagu
WORKDIR /home/dagu
RUN <<EOF
    #dagu binary setup
    if [ "${TARGETARCH}" == "amd64" ]; then 
        arch="x86_64";
    else 
        arch="${TARGETARCH}"
    fi
    export TARGET_FILE="dagu_${VERSION}_Linux_${arch}.tar.gz"
    wget ${RELEASES_URL}/download/v${VERSION}/${TARGET_FILE}
    tar -xf ${TARGET_FILE} && rm *.tar.gz 
    sudo mv dagu /usr/local/bin/ 
    mkdir .dagu
EOF

ENV DAGU_HOST=0.0.0.0
ENV DAGU_PORT=8080

CMD dagu server

To build the image, you can simply run the following command:

DAGU_VERSION=1.10.4
docker build -t dagu:${DAGU_VERSION} \
--platform linux/amd64 \
--build-arg VERSION=${DAGU_VERSION} \
--no-cache .

Let me know if you have any questions or if there's anything else I can do to assist you. Thanks for your help in resolving this issue!

[rajasekhar-gundala]

Hi @rajaseg,

I'm sorry to hear that the image isn't working on your computer. Would you mind building the Docker image on your end with the following Dockerfile? It should help us determine the issue and get it resolved.

Dockerfile example:

FROM --platform=$BUILDPLATFORM alpine:latest

ARG TARGETARCH
ARG VERSION=1.3.15 
ARG RELEASES_URL="https://github.com/yohamta/dagu/releases"

ARG USER="dagu"
ARG USER_UID=1000
ARG USER_GID=$USER_UID

EXPOSE 8080

RUN <<EOF
    #User and permissions setup
    apk update
    apk add --no-cache sudo tzdata
    addgroup -g ${USER_GID} ${USER}
    adduser ${USER} -h /home/${USER} -u ${USER_UID} -G ${USER} -D -s /bin/ash
    echo ${USER} ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/${USER}
    chmod 0440 /etc/sudoers.d/${USER}
EOF

USER dagu
WORKDIR /home/dagu
RUN <<EOF
    #dagu binary setup
    if [ "${TARGETARCH}" == "amd64" ]; then 
        arch="x86_64";
    else 
        arch="${TARGETARCH}"
    fi
    export TARGET_FILE="dagu_${VERSION}_Linux_${arch}.tar.gz"
    wget ${RELEASES_URL}/download/v${VERSION}/${TARGET_FILE}
    tar -xf ${TARGET_FILE} && rm *.tar.gz 
    sudo mv dagu /usr/local/bin/ 
    mkdir .dagu
EOF

ENV DAGU_HOST=0.0.0.0
ENV DAGU_PORT=8080

CMD dagu server

To build the image, you can simply run the following command:

DAGU_VERSION=1.10.4
docker build -t dagu:${DAGU_VERSION} \
--platform linux/amd64 \
--build-arg VERSION=${DAGU_VERSION} \
--no-cache .

Let me know if you have any questions or if there's anything else I can do to assist you. Thanks for your help in resolving this issue!

Hey @yohamta I think the issue with the below line in Dockerfile

ARG VERSION=1.3.15

I replaced it with ARG VERSION=1.10.4 and built the Docker Container I was able to deploy it and access the Web UI.

Coming to the actual part (running Meltano ETL) I tried with the below DAG. But I got below error.

steps:
  - name: step1
    command: meltano elt tap-github target-jsonl

In order to run the meltano commands, we need to install meltano, create meltano.yml file which contains the details about the extractor and loader.

https://docs.meltano.com/getting-started/installation

[yohamta]

Hey @rajaseg! It's fantastic to hear that you can run the image now.

As for Meltano ETL, I'm not entirely certain about the command to use Meltano. The example I provided earlier was just a pseudo command. Dagu is essentially a tool to execute commands based on DAG definition and nothing more. I'd recommend taking a look at Meltano's documentation to learn how to run the commands properly. Good luck, and let me know if you have any other questions!

[luo-geng]

I also encountered this issue and found an article

[yohamta]

I have just fixed the issue. Sorry for taking a long time.