mxkrn
commented
1 year ago Generally this already is a great starting point, I'd like to make a few smaller suggestions that I think would be nice to work into this refactor.
- I think we're currently not rigorously checking code style or type annotations. Let's have a round of running those scripts and making sure these are all valid
-
21 Suggests some basic CI for enforcing the above. We can use GitHub actions in the same fashion other repos use it. I would suggest to avoid running the tests in CI for now because it requires a pretty complicated environment setup.
- Let's tackle #8 in order to avoid the
klay_beampackage st the lean, universal package that we want it to be - If we get rid of it
torchaudio(which I think is the easiest way to remove thetorchdependency fromklay_beam) then we can also close #7 -
6 is a great idea, it's also very simple to implement named tuples.
- We can probably manage our versioning process as it relates to
klay_beamand the docker versio a bit better, this issue describes this in greater detail
CharlesHolbrow
Getting to Klay Beam v1.0
Right now, the main function of the repo is to demonstrate HOW to do things in Beam. Now that we know how Beam works, let's think about is to refactor the core functionality into a package, and develop reusable operational processes for creating and running data pipelines that depend on that package.
v1.0 should properly separate the package logic from the operational logic.
The Klay Beam Package
The core functionality that we tend to reuse in many jobs (Is this list missing anything?):
This behavior could be moved into a lightweight package. Additional job-speciffic Transforms should be in a dedicated package or launch script.
The Klay Beam Operational Processes
This examples in this repo show how to do a lot of complex jobs. In isolation the job configuration options below are simple enough, but we've found that different jobs tend to need very precise combination of them.
apache_beampython packageWe'd like to be able to easily use these examples in this repo create, test, and run jobs with various combinations of these configuration options.
Message Passing Conventions
Moving this to a dedicated issue:
41
Job packages
As Max proposed in #18, let's cleanup Dockerfiles
Each job should have its own environment and and pin a docker image. If it helps, we can make a parent Dockerfile for large Dependencies and make job-specific Docker images 'FROM' the parent.
GPU Processing Support
Moving this to a dedicated issue:
67
Related Issues
v1.0 should resolve these issues
6
18
31