Basic Hasura-based API as a service

rufuspollock commented 4 years ago

Epic here #1

When developing data-api I want to be able to create and run integration tests of the app so that I can check how it's working together with postgres and hasura.

Acceptance

[x] There is a docker-compose environment with the necessary cervices
[x] There is ci running tests and publishing to a Docker repository

Tasks

Local developer docker-compose

[x] add services to docker-compose
- [x] postres (latest official postgres image)
- [x] hasura (custom image with adding sample data)
[x] write a basic smoke test checking the system parts are working
- [x] Find the framework to use (Supertest remains the default for node http server tests ...)

CI

[x] Dockerfile (FROM nodejs ... + copy files)
[x] Publish Docker image (used https://github.com/datopian/data-api/new/master?filename=.github%2Fworkflows%2Fdocker-publish.yml&workflow_template=docker-publish for how to)
[x] Add another docker-compose for ci
- [x] add data-api image building from source (include tests)
[x] check code correctness in CI on every push:
- [x] run prettier
- [x] run tests
[ ] create new version Docker tags automatically - CI on every push to master:
- [ ] update version (if tests ok)
- [ ] publish new version of the image
- [x] publish :latest tag with the new changes

Inbox

[ ] Have Hasura based read API

[ ] Dockerize this service and auto publish on merge to master

[ ] Work out where we publish to (is it dockerhub or is it github packages? (Ask on tech help?)

[ ] Publish there with continuous deployment (on master)

[ ] Testing

[ ] Have basic test for this (using cypress (?))

[ ] CI setup and passing

EvgeniiaVak commented 3 years ago

@rufuspollock I don't think we really need to store hasura settings anywhere or publish a custom hasura image, because what we will set up in hasura is

mark which tables we want to track (hasura metadata)
run any sql scripts preparing tables/views (hasura migrations)

That is very project specific, I don't think we can have anything Datopian wide here. We can just launch hasura service in a project (like we would launch postgres):

Example Hasura Dockerfile

FROM hasura/graphql-engine:latest.cli-migrations-v2

COPY migrations /hasura-migrations/
COPY metadata /hasura-metadata/

Example docker-compose containing Postgres, Hasura, A third service

version: '3'
services:
  wrapper:
    image: my-app-wrapper
    build:
      context: wrapper/
      dockerfile: Dockerfile

  graphql-engine:
    image: my-app-hasura
    build:
      context: hasura/
      dockerfile: Dockerfile
    ports:
      - "8080:8080"
    depends_on:
      - db
    restart: always
    environment:
      HASURA_GRAPHQL_DATABASE_URL: postgres://ckan_default:password@db:5432/datastore_default
      ## enable the console served by server
      HASURA_GRAPHQL_ENABLE_CONSOLE: "true" # set to "false" to disable console
      ## enable debugging mode. It is recommended to disable this in production
      HASURA_GRAPHQL_DEV_MODE: "true"
      HASURA_GRAPHQL_ENABLED_LOG_TYPES: startup, http-log, webhook-log, websocket-log, query-log
      ## uncomment next line to set an admin secret
      # HASURA_GRAPHQL_ADMIN_SECRET: myadminsecretkey
  db:
    image: my-app-postgres
    build:
      context: postgres/
      dockerfile: Dockerfile
    ports:
      - "5434:5432"
    environment:
      POSTGRES_DB: datastore_default
      POSTGRES_USER: ckan_default
      POSTGRES_PASSWORD: password

rufuspollock commented 3 years ago

Agreed

On Wed, 9 Sep 2020, 16:20 EvgeniiaVak, notifications@github.com wrote:

@rufuspollock https://github.com/rufuspollock I don't think we really need to store hasura settings anywhere or publish a custom hasura image, because what we will set up in hasura is

mark which tables we want to track (hasura metadata https://hasura.io/docs/1.0/graphql/core/migrations/manage-metadata.html#manage-hasura-metadata )

run any sql scripts preparing tables/views (hasura migrations https://hasura.io/docs/1.0/graphql/core/migrations/migrations-setup.html#migrations-setup )

That is very project specific, I don't think we can have anything Datopian wide here. We can just launch hasura service in a project (like we would launch postgres): Example Hasura Dockerfile

FROM hasura/graphql-engine:latest.cli-migrations-v2 COPY migrations /hasura-migrations/COPY metadata /hasura-metadata/

Example docker-compose containing Postgres, Hasura, A third service

version: '3'services: wrapper: image: my-app-wrapper build: context: wrapper/ dockerfile: Dockerfile

graphql-engine: image: my-app-hasura build: context: hasura/ dockerfile: Dockerfile ports:

"8080:8080" depends_on:

db restart: always environment: HASURA_GRAPHQL_DATABASE_URL: postgres://ckan_default:password@db:5432/datastore_default
enable the console served by server

HASURA_GRAPHQL_ENABLE_CONSOLE: "true" # set to "false" to disable console

enable debugging mode. It is recommended to disable this in production

HASURA_GRAPHQL_DEV_MODE: "true" HASURA_GRAPHQL_ENABLED_LOG_TYPES: startup, http-log, webhook-log, websocket-log, query-log

uncomment next line to set an admin secret

HASURA_GRAPHQL_ADMIN_SECRET: myadminsecretkey

db: image: my-app-postgres build: context: postgres/ dockerfile: Dockerfile ports:

"5434:5432" environment: POSTGRES_DB: datastore_default POSTGRES_USER: ckan_default POSTGRES_PASSWORD: password

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/datopian/data-api/issues/4#issuecomment-689594881, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABMDMWEFVM3ZZIG3ZP24STSE6FLJANCNFSM4Q767G5A .

EvgeniiaVak commented 3 years ago

WON'T FIX. See above comments.

rufuspollock commented 3 years ago

@EvgeniiaVak what i would be doing here is listing the docker compose setup you have described in e.g. the README and actually setting that up for this project with CI - so we have a basic test for running hasura against a sample DB (i know this may duplicate some work elsewhere but doing it here should be trivial and is useful IMO).

EvgeniiaVak commented 3 years ago

Tests are passing in docker runner:

https://hub.docker.com/repository/registry-1.docker.io/datopian/data-api/builds/07692fac-c5db-407d-ace1-2c6ea400244b

...
$ mocha -r dotenv/config --recursive
data-api
[0mGET /non_existing_page [33m404 [0m4.439 ms - 10[0m
✓ should return 404 for non existing page (40ms)
GraphQL endpoint
[0mPOST /v1/graphql [32m200 [0m25.855 ms - -[0m
✓ returns graphql schema (61ms)
2 passing (112ms)
...

EvgeniiaVak commented 3 years ago

Prettier check is running on new builds:

https://hub.docker.com/repository/registry-1.docker.io/datopian/data-api/builds/3ca791c7-b76c-4461-a814-2afa40045a43

Step 9/10 : RUN ["yarn", "prettier", "--check", "."]
---> Running in 79016bfb81d4
yarn run v1.22.4
$ /usr/src/app/node_modules/.bin/prettier --check .
Checking formatting...
All matched files use Prettier code style!

rufuspollock commented 3 years ago

@EvgeniiaVak great - can we tick acceptance then and close?

EvgeniiaVak commented 3 years ago

@rufuspollock let's not close it just yet, I really want to make this automatic version tagging work (right now the it's always data-api:latest), also github seems to run tests too, and they fail (I think additionally to dockerhub (which pass))

EvgeniiaVak commented 3 years ago

When trying to push a change to the github workflow file (.github/workflows/docker-publish.yml) from local machine there is an error:

evgeniyavakarina (master) data-api $ git push
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 4 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (5/5), 624 bytes | 624.00 KiB/s, done.
Total 5 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To https://github.com/datopian/data-api.git
 ! [remote rejected] master -> master (refusing to allow an OAuth App to create or update workflow `.github/workflows/docker-publish.yml` without `workflow` scope)
error: failed to push some refs to 'https://github.com/datopian/data-api.git'

Using GitHub UI for that for now as a solution.

EvgeniiaVak commented 3 years ago

@zelima I think I need your help here.

I created a github action (as from this example https://github.com/datopian/data-api/new/master?filename=.github%2Fworkflows%2Fdocker-publish.yml&workflow_template=docker-publish) - .github/workflows/docker-publish.yml but tests always failed because I couldn't quite set up environment on github correctly, so I deleted it.

On dockerhub tests always run (even without ci on github) and they run successfully example of the last one - https://hub.docker.com/repository/registry-1.docker.io/datopian/data-api/builds/d4993ba9-d577-413c-aa17-bfcd9f3d4705 (it's without any ci set up in github already).

What I think was happening: both github and dockerhub have ci running (I guess in dockerhub it's the one you've set up during our call @zelima ).

What I think should be happening: only github ci running:

tests
updating app version and creating a corresponding docker image tag (if tests passed)
publishin :latest and vX tagged images on dockerhub

Maybe I'm wrong here and only dockerhub ci should be running. Please let me know if we have any examples at Datopian already set up correctly so I could create a similar setup.

zelima commented 3 years ago

@EvgeniiaVak I'm not sure about the details. But probably docker is not running any tests, unless we somehow tell it. Ther eporbably is a way the we did not setup. So my guess is it'sonly running the build part.

I'm can't remember any Datopian project that does that

EvgeniiaVak commented 3 years ago

@zelima no, it's running the tests: e.g. see this one https://hub.docker.com/repository/registry-1.docker.io/datopian/data-api/builds/d4993ba9-d577-413c-aa17-bfcd9f3d4705. it has all the logs from mocha tests:

$ mocha -r dotenv/config --recursive
data-api
[0mGET /non_existing_page [33m404 [0m4.667 ms - 10[0m
✓ should return 404 for non existing page (43ms)
GraphQL endpoint
[0mPOST /v1/graphql [32m200 [0m28.146 ms - -[0m
✓ returns graphql schema (64ms)
2 passing (123ms)
Done in 0.79s.

here is the documentation for it - https://docs.docker.com/docker-hub/builds/automated-testing/

And it would be perfect to live it like this (since ci runs as expected but on dockerhub, rather than github) but I don't see a way to create version tags for images there...

EvgeniiaVak commented 3 years ago

@rufuspollock I think we should close this issue and create another additional one.

What we have now - every time there is a push to master branch tests run on dockerhub and if pass a new docker image (tagged :latest) is built (I think it meets the criteria from the inbox).

I think we still need versioning but since it's not so straightforward let's create a separate issue. And another nice thing would be to have tests running on pull requests to master also.

zelima commented 3 years ago

@EvgeniiaVak I've just looked in docker build configurations and seems like you can configure it so that it builds from tags.

That's on docker side. From github, you can either simply push a new release/tag manually. Or maybe even write small script to run that with travis.

zelima commented 3 years ago

Also just remembered think ckan-cloud-helm is deploying to dockerhub after succefull tests https://github.com/datopian/ckan-cloud-helm/blob/master/.travis.yml

EvgeniiaVak commented 3 years ago

Thank you @zelima that's really helpful!

So if we go with builds from tags, we would still need to create those tags first (manually?)... 🤔

zelima commented 3 years ago

So if we go with builds from tags, we would still need to create those tags first (manually?)...

Yes. Usually, that's done manually. But think you can easily scrip it if you really want. Or maybe even github has some tools to do that for you

EvgeniiaVak commented 3 years ago

Ok. Tests on pull requests are also running (without building a new image)! 🎉

I created a test PR from branch test-ci. Pushed a commit that adds a file. Then pushed a commit that deletes the test file. On the page https://hub.docker.com/repository/docker/datopian/data-api/builds. We can see 2 builds from test-ci branch:

EvgeniiaVak commented 3 years ago

FIXED. Test setup is in docker-compose.test.yml file (as per Dockerhub docs https://docs.docker.com/docker-hub/builds/automated-testing/). CI jobs here https://hub.docker.com/repository/docker/datopian/data-api/builds.

datopian / data-api