Linking to container image via --template when creating AZ Batch pool

gvegayon commented 1 month ago

This pull request includes significant updates to the Azure Batch Pool setup process, including adding a new Python script for pool creation and configuration and updates to the workflow and dependencies.

Azure Batch Pool Setup Enhancements:

Added a new Python script azure/pool.py to handle Azure Batch Pool creation and configuration using a TOML configuration file. This script includes functions for creating containers, reading autoscale formulas, and setting up the batch pool with the necessary credentials and configurations.
Updated the workflow file .github/workflows/containers-and-az-pool.yaml to run the new Python script, replacing the previous direct Azure CLI commands. This includes steps for writing the configuration file, ensuring the Azure CLI is installed, and running the Python script. [1] [2] [3]

Documentation and Dependency Updates:

Updated NEWS.md to include a note about the new script for setting up the Azure Batch Pool.
Added new dependencies to azure/requirements.txt required by the new Python script, including azure-common, azure-core, azure-identity, azure-keyvault, azure-mgmt-batch, and toml.

codecov[bot] commented 1 month ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Additional details and impacted files

:loudspeaker: Thoughts on this report? Let us know!

🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests

github-actions[bot] commented 1 week ago

Thank you for your contribution @gvegayon :rocket:! Your pkgdown-site is ready for download :point_right: here :point_left:! _{(The artifact expires on 2024-11-29T16:25:03Z. You can re-generate it by re-running the workflow here.)}

gvegayon commented 1 week ago

So we are getting closer @jkislin and @zsusswein. There's an issue with the generated toml file that is makes it fail during the read process:

    config = toml.load(config_file)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/toml/decoder.py", line 134, in load
    return loads(ffile.read(), _dict, decoder)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/toml/decoder.py", line 514, in loads
    raise TomlDecodeError(str(err), original, pos)
toml.decoder.TomlDecodeError: Found tokens after a closed string. Invalid TOML. (line 10 column 1 char 754)

I'll try to debug it tomorrow, but I think we are almost there!

gvegayon commented 1 day ago

I am getting errors on testthat b/c of duckdb. The issue is intermittent, so either we are missing a seed, or there's something else going on. https://github.com/CDCgov/cfa-epinow2-pipeline/actions/runs/11943535928/job/33293099360#step:3:309. I have had to restart the job many times in the past, with the second try working. Unsure what's the issue

attn @natemcintosh @zsusswein

zsusswein commented 1 day ago

That's weird. I've seen an intermittently flaky test from DuckDB but it's rare and I thought I had resolved it.

Can you open an issue for it and I'll look into it?

jkislin commented 22 hours ago

@gvegayon @zsusswein do we want to merge this now?

zsusswein commented 22 hours ago

Is the PR ready to go? I had been assuming it was waiting for something because it was still draft.

gvegayon commented 2 hours ago

Given that the tests pass, this is ready for review @zsusswein and @jkislin!

CDCgov / cfa-epinow2-pipeline