WIP: docker image testing

[td928]

352 still a work in progress

recall the lack of functionality to test the staging docker image. This is an attempt to get at that by creating a dummy template to run a datasets.

But you can also see that by adding the subfolder options under destination here:

  destination:
    name: *name
    subfolder: test_docker_

and some modification in the ingestor function it could achieved such that one can dictate using the template to whether send any datasets to their default folder or under a "subfolder" which in this case is test_docker_. This additional layer solves our issue of adding something potentially only for testing to our production datasets folders.

Next Steps

This in my mind could eventually be formalized into a workflow where a data matrix could be iterated over like we do in the weekly update workflow and then passing the subfolder as an option like:

jobs:
  dataloading:
    runs-on: ubuntu-latest
    env:
      AWS_S3_ENDPOINT: ${{ secrets.DO_S3_ENDPOINT }}
      AWS_ACCESS_KEY_ID: ${{ secrets.DO_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.DO_SECRET_ACCESS_KEY }}
      AWS_S3_BUCKET: edm-recipes
    steps:
      - uses: actions/checkout@v2.3.4
      - uses: NYCPlanning/action-library-archive@v1.2
        name: Archive test_docker
        with:
          name: ${{ github.event.inputs.dataset }}
          subfolder: test
          latest: ${{ github.event.inputs.latest }}
          version: ${{ github.event.inputs.version }}
          s3: true
          compress: true
          output_format: pgdump shapefile csv

then it would send a few datasets for testing to the test folder on digital ocean for review.