DAGWorks-Inc / hamilton

Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
https://hamilton.dagworks.io/en/latest/
BSD 3-Clause Clear License
1.88k stars 126 forks source link

Provide dockerx setup for docker builds #1194

Closed sikehish closed 1 month ago

sikehish commented 1 month ago

This address #1064. A shell script was created to handle building and pushing multi-architecture images using docker buildx.

Changes

This PR introduces a shell script (buildx_and_push.sh) to automate the process of building and pushing Docker images for multiple architectures using docker buildx. It ensures that the Docker images are always built with the latest version of the sf-hamilton-ui library from PyPI.

Key Changes:

  1. Multi-architecture support:

    • Builds images for multiple platforms (linux/amd64, linux/arm64).
  2. Automated version management:

    • The script dynamically fetches the latest sf-hamilton-ui version from PyPI.
    • The fetched version is used to tag Docker images, ensuring that both the versioned tag and latest tag are always in sync with the current release.
  3. Builder instance setup:

    • Ensures that Docker Buildx is enabled and uses a named builder instance (hamilton-builder).
  4. Image tagging and pushing:

    • Tags both backend and frontend images with:
      • The latest sf-hamilton-ui version.
      • The latest tag.

How I Tested This

  1. Environment:

    • Tested on a local machine with Docker Buildx enabled.
  2. Test Cases:

    • Checked the output to confirm that the fetched version from PyPI was used for tagging the images.
    • Built and pushed the backend and frontend images successfully using:
      ./buildx_and_push.sh
    • Verified that both versioned and latest tags were pushed to the repository.
  3. Validation:

    • Confirmed that the following tags were pushed:
      • dagworks/ui-backend:<fetched_version>
      • dagworks/ui-backend:latest
      • dagworks/ui-frontend:<fetched_version>
      • dagworks/ui-frontend:latest

Notes

sikehish commented 1 month ago

Looks great!

what would a github action look like to get this to run once a new sf-hamilton-ui version was published? I'll be working on the action soon. But I'm currently building the images without the env variables/args. What do I do? Do I add multi arch builds to the docker compose file, or is there any other approach that I need to take?

skrawcz commented 1 month ago

I'll be working on the action soon. But I'm currently building the images without the env variables/args. What do I do? Do I add multi arch builds to the docker compose file, or is there any other approach that I need to take?

Sorry I'm not following. The multi-arch builds would just use your script here right?

sikehish commented 1 month ago

I'll be working on the action soon. But I'm currently building the images without the env variables/args. What do I do? Do I add multi arch builds to the docker compose file, or is there any other approach that I need to take?

Sorry I'm not following. The multi-arch builds would just use your script here right?

Yes. Just wanted to ask if you if the script is satisfying the requirements.

skrawcz commented 1 month ago

Yes. Just wanted to ask if you if the script is satisfying the requirements.

Script looks great thanks. I haven't run it yet to verify though; so I assume it works right now.

sikehish commented 1 month ago

Looks great!

what would a github action look like to get this to run once a new sf-hamilton-ui version was published?

Looks great!

what would a github action look like to get this to run once a new sf-hamilton-ui version was published?

Hi @skrawcz . Ive created a new workflow, tested it on my fork, and it's building and pushing those images. So it's working fine. Let me know if I need to make any changes :)

sikehish commented 1 month ago

Hi @skrawcz I've made the reuqested changes. Let me know if anything else needs to be changed. I've tested it on my local fork, and its working fine.

sikehish commented 1 month ago

just tried it - seems to work - i just had to change two small things

Oh Okay. Anything else that I need to work on?

skrawcz commented 1 month ago

@sikehish just need to fix up the whitespace pre-commit issues. Otherwise this is ready to merge. Thanks!

e.g. pip install pre-commit pre-commit run --all-files I think those are the commands

sikehish commented 1 month ago

@sikehish just need to fix up the whitespace pre-commit issues. Otherwise this is ready to merge. Thanks!

e.g. pip install pre-commit pre-commit run --all-files I think those are the commands

I have executed the above command. Does everything look good now? Thank you for the opportunity:)) It was fun working on this issue.