Closed rufuspollock closed 3 years ago
@rufuspollock I don't think we really need to store hasura settings anywhere or publish a custom hasura image, because what we will set up in hasura is
That is very project specific, I don't think we can have anything Datopian wide here. We can just launch hasura service in a project (like we would launch postgres):
FROM hasura/graphql-engine:latest.cli-migrations-v2
COPY migrations /hasura-migrations/
COPY metadata /hasura-metadata/
version: '3'
services:
wrapper:
image: my-app-wrapper
build:
context: wrapper/
dockerfile: Dockerfile
graphql-engine:
image: my-app-hasura
build:
context: hasura/
dockerfile: Dockerfile
ports:
- "8080:8080"
depends_on:
- db
restart: always
environment:
HASURA_GRAPHQL_DATABASE_URL: postgres://ckan_default:password@db:5432/datastore_default
## enable the console served by server
HASURA_GRAPHQL_ENABLE_CONSOLE: "true" # set to "false" to disable console
## enable debugging mode. It is recommended to disable this in production
HASURA_GRAPHQL_DEV_MODE: "true"
HASURA_GRAPHQL_ENABLED_LOG_TYPES: startup, http-log, webhook-log, websocket-log, query-log
## uncomment next line to set an admin secret
# HASURA_GRAPHQL_ADMIN_SECRET: myadminsecretkey
db:
image: my-app-postgres
build:
context: postgres/
dockerfile: Dockerfile
ports:
- "5434:5432"
environment:
POSTGRES_DB: datastore_default
POSTGRES_USER: ckan_default
POSTGRES_PASSWORD: password
Agreed
On Wed, 9 Sep 2020, 16:20 EvgeniiaVak, notifications@github.com wrote:
@rufuspollock https://github.com/rufuspollock I don't think we really need to store hasura settings anywhere or publish a custom hasura image, because what we will set up in hasura is
- mark which tables we want to track (hasura metadata https://hasura.io/docs/1.0/graphql/core/migrations/manage-metadata.html#manage-hasura-metadata )
- run any sql scripts preparing tables/views (hasura migrations https://hasura.io/docs/1.0/graphql/core/migrations/migrations-setup.html#migrations-setup )
That is very project specific, I don't think we can have anything Datopian wide here. We can just launch hasura service in a project (like we would launch postgres): Example Hasura Dockerfile
FROM hasura/graphql-engine:latest.cli-migrations-v2 COPY migrations /hasura-migrations/COPY metadata /hasura-metadata/
Example docker-compose containing Postgres, Hasura, A third service
version: '3'services: wrapper: image: my-app-wrapper build: context: wrapper/ dockerfile: Dockerfile
graphql-engine: image: my-app-hasura build: context: hasura/ dockerfile: Dockerfile ports:
- "8080:8080" depends_on:
- db restart: always environment: HASURA_GRAPHQL_DATABASE_URL: postgres://ckan_default:password@db:5432/datastore_default
enable the console served by server
HASURA_GRAPHQL_ENABLE_CONSOLE: "true" # set to "false" to disable console
enable debugging mode. It is recommended to disable this in production
HASURA_GRAPHQL_DEV_MODE: "true" HASURA_GRAPHQL_ENABLED_LOG_TYPES: startup, http-log, webhook-log, websocket-log, query-log
uncomment next line to set an admin secret
HASURA_GRAPHQL_ADMIN_SECRET: myadminsecretkey
db: image: my-app-postgres build: context: postgres/ dockerfile: Dockerfile ports:
- "5434:5432" environment: POSTGRES_DB: datastore_default POSTGRES_USER: ckan_default POSTGRES_PASSWORD: password
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/datopian/data-api/issues/4#issuecomment-689594881, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABMDMWEFVM3ZZIG3ZP24STSE6FLJANCNFSM4Q767G5A .
WON'T FIX. See above comments.
@EvgeniiaVak what i would be doing here is listing the docker compose setup you have described in e.g. the README and actually setting that up for this project with CI - so we have a basic test for running hasura against a sample DB (i know this may duplicate some work elsewhere but doing it here should be trivial and is useful IMO).
Tests are passing in docker runner:
...
$ mocha -r dotenv/config --recursive
data-api
[0mGET /non_existing_page [33m404 [0m4.439 ms - 10[0m
✓ should return 404 for non existing page (40ms)
GraphQL endpoint
[0mPOST /v1/graphql [32m200 [0m25.855 ms - -[0m
✓ returns graphql schema (61ms)
2 passing (112ms)
...
Prettier check is running on new builds:
Step 9/10 : RUN ["yarn", "prettier", "--check", "."]
---> Running in 79016bfb81d4
yarn run v1.22.4
$ /usr/src/app/node_modules/.bin/prettier --check .
Checking formatting...
All matched files use Prettier code style!
@EvgeniiaVak great - can we tick acceptance then and close?
@rufuspollock let's not close it just yet, I really want to make this automatic version tagging work (right now the it's always data-api:latest), also github seems to run tests too, and they fail (I think additionally to dockerhub (which pass))
When trying to push a change to the github workflow file (.github/workflows/docker-publish.yml
) from local machine there is an error:
evgeniyavakarina (master) data-api $ git push
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 4 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (5/5), 624 bytes | 624.00 KiB/s, done.
Total 5 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To https://github.com/datopian/data-api.git
! [remote rejected] master -> master (refusing to allow an OAuth App to create or update workflow `.github/workflows/docker-publish.yml` without `workflow` scope)
error: failed to push some refs to 'https://github.com/datopian/data-api.git'
Using GitHub UI for that for now as a solution.
@zelima I think I need your help here.
I created a github action (as from this example https://github.com/datopian/data-api/new/master?filename=.github%2Fworkflows%2Fdocker-publish.yml&workflow_template=docker-publish) - .github/workflows/docker-publish.yml
but tests always failed because I couldn't quite set up environment on github correctly, so I deleted it.
On dockerhub tests always run (even without ci on github) and they run successfully example of the last one - https://hub.docker.com/repository/registry-1.docker.io/datopian/data-api/builds/d4993ba9-d577-413c-aa17-bfcd9f3d4705 (it's without any ci set up in github already).
What I think was happening: both github and dockerhub have ci running (I guess in dockerhub it's the one you've set up during our call @zelima ).
What I think should be happening: only github ci running:
:latest
and vX
tagged images on dockerhubMaybe I'm wrong here and only dockerhub ci should be running. Please let me know if we have any examples at Datopian already set up correctly so I could create a similar setup.
@EvgeniiaVak I'm not sure about the details. But probably docker is not running any tests, unless we somehow tell it. Ther eporbably is a way the we did not setup. So my guess is it'sonly running the build part.
I'm can't remember any Datopian project that does that
@zelima no, it's running the tests: e.g. see this one https://hub.docker.com/repository/registry-1.docker.io/datopian/data-api/builds/d4993ba9-d577-413c-aa17-bfcd9f3d4705. it has all the logs from mocha
tests:
$ mocha -r dotenv/config --recursive
data-api
[0mGET /non_existing_page [33m404 [0m4.667 ms - 10[0m
✓ should return 404 for non existing page (43ms)
GraphQL endpoint
[0mPOST /v1/graphql [32m200 [0m28.146 ms - -[0m
✓ returns graphql schema (64ms)
2 passing (123ms)
Done in 0.79s.
here is the documentation for it - https://docs.docker.com/docker-hub/builds/automated-testing/
And it would be perfect to live it like this (since ci runs as expected but on dockerhub, rather than github) but I don't see a way to create version tags for images there...
@rufuspollock I think we should close this issue and create another additional one.
What we have now - every time there is a push to master branch tests run on dockerhub and if pass a new docker image (tagged :latest
) is built (I think it meets the criteria from the inbox).
I think we still need versioning but since it's not so straightforward let's create a separate issue. And another nice thing would be to have tests running on pull requests to master also.
@EvgeniiaVak I've just looked in docker build configurations and seems like you can configure it so that it builds from tags.
That's on docker side. From github, you can either simply push a new release/tag manually. Or maybe even write small script to run that with travis.
Also just remembered think ckan-cloud-helm is deploying to dockerhub after succefull tests https://github.com/datopian/ckan-cloud-helm/blob/master/.travis.yml
Thank you @zelima that's really helpful!
So if we go with builds from tags, we would still need to create those tags first (manually?)... 🤔
So if we go with builds from tags, we would still need to create those tags first (manually?)...
Yes. Usually, that's done manually. But think you can easily scrip it if you really want. Or maybe even github has some tools to do that for you
Ok. Tests on pull requests are also running (without building a new image)! 🎉
I created a test PR from branch test-ci
. Pushed a commit that adds a file. Then pushed a commit that deletes the test file. On the page https://hub.docker.com/repository/docker/datopian/data-api/builds. We can see 2 builds from test-ci
branch:
FIXED. Test setup is in docker-compose.test.yml
file (as per Dockerhub docs https://docs.docker.com/docker-hub/builds/automated-testing/). CI jobs here https://hub.docker.com/repository/docker/datopian/data-api/builds.
Epic here #1
When developing data-api I want to be able to create and run integration tests of the app so that I can check how it's working together with postgres and hasura.
Acceptance
Tasks
Local developer docker-compose
CI
FROM nodejs ...
+ copy files)Inbox