search

thegraphnetwork / EpiGraphHub

Software platform to Gather, transmform, harmonize and store epidemiological data for analytical purposes.
https://epigraphhub.org
GNU General Public License v3.0
8 stars 10 forks source link

feat(tests): add tests to EpiGraphHub webserver #183

Closed luabida closed 1 year ago

luabida commented 1 year ago

Tests include:

fix #179

xmnlab commented 1 year ago

I will provide some examples in a bit

On Wed, Apr 5, 2023, 13:13 Luã Bida Vacaro @.***> wrote:

@.**** commented on this pull request.

In tests/test_webapp.py https://github.com/thegraphnetwork/EpiGraphHub/pull/183#discussion_r1158799261 :

@@ -0,0 +1,40 @@ +from selenium import webdriver +from selenium.webdriver.common.by import By +import unittest +import os + +CHROMEDRIVER = os.getenv('CHROMEDRIVER') + +class TestEpiGraphHub(unittest.TestCase):

  • @classmethod
  • def setUpClass(cls):
  • cls.driver = webdriver.Chrome(CHROMEDRIVER)
  • cls.driver.get("https://epigraphhub.org/")

I'm aware, and the step should be below the containers initialization. But the chromedriver nor chrome-setup seem to be working on CI, kinda lost at this point

— Reply to this email directly, view it on GitHub https://github.com/thegraphnetwork/EpiGraphHub/pull/183#discussion_r1158799261, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHX5HNNCBFLEOCICD3HSF3W7WR3XANCNFSM6AAAAAAWUAXBNE . You are receiving this because your review was requested.Message ID: @.***>

fccoelho commented 1 year ago

Thanks for pushing this @xmnlab, this will be very important for the maintenability of the egh services

luabida commented 1 year ago

I will provide some examples in a bit

@xmnlab the last solution I have in mind is instead of using the chromedriver action, install the chrome with wget and run it as a step, any thoughts about trying this approach?

xmnlab commented 1 year ago

maybe you can find something here: https://github.com/osl-pocs/poc-python-selenium/

luabida commented 1 year ago

@xmnlab it seems like the postgres container is not initializing properly, dor you have any idea why the postgres volumes of the compose-dev are not working correctly?

ref: https://github.com/thegraphnetwork/EpiGraphHub/actions/runs/4629712850/jobs/8190327546#step:14:2951

xmnlab commented 1 year ago

I don't think that it is a volume problem the volume is not pointing to a specific path in the host side: https://github.com/thegraphnetwork/EpiGraphHub/blob/main/containers/compose-dev.yaml#L18

but if you want to test, you can remove that line and check how it would be performed by the CI

luabida commented 1 year ago

@xmnlab the file that has the line GRANT ALL ON SCHEMA switzerland TO dev_epigraph; is one of the volumes below that, more specifically the ./postgresql/sql/dev/02-epigraphhub.sql

xmnlab commented 1 year ago

Got it .. have you tried that locally as well? Just to check where is the problem .. maybe a new version of the image changed the path etc

luabida commented 1 year ago

@xmnlab Yes, the same error occurs locally

xmnlab commented 1 year ago

@xmnlab Yes, the same error occurs locally

can you share what you have in the logs? maybe it would give us some tips

PS: btw, in the worst case we could have it as a COPY statement in the dockerfile

luabida commented 1 year ago

@xmnlab can you share what you have in the logs? maybe it would give us some tips

sure, the logs are very similar

[2023-04-06, 17:47:18 UTC] {{__init__.py:117}} DEBUG - Preparing lineage inlets and outlets [2023-04-06, 17:47:18 UTC] {{__init__.py:155}} DEBUG - inlets: [], outlets: [] [2023-04-06, 17:47:18 UTC] {{taskinstance.py:1775}} ERROR - Task failed with exception Traceback (most recent call last): File "/opt/conda/envs/epigraphhub/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context self.dialect.do_execute( File "/opt/conda/envs/epigraphhub/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute cursor.execute(statement, parameters) psycopg2.errors.InsufficientPrivilege: permission denied for schema switzerland LINE 1: select * from switzerland.foph_casesvaccpersons_d; ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/conda/envs/epigraphhub/lib/python3.9/site-packages/airflow/operators/python.py", line 210, in execute branch = super().execute(context) File "/opt/conda/envs/epigraphhub/lib/python3.9/site-packages/airflow/operators/python.py", line 175, in execute return_value = self.execute_callable() File "/opt/conda/envs/epigraphhub/lib/python3.9/site-packages/airflow/operators/python.py", line 192, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/opt/airflow/dags/foph_dag.py", line 204, in compare if not loading.compare(filename, tablename): File "/opt/conda/envs/epigraphhub/lib/python3.9/site-packages/epigraphhub/data/foph/loading.py", line 68, in compare table_date = _table_last_update(table) File "/opt/conda/envs/epigraphhub/lib/python3.9/site-packages/epigraphhub/data/foph/loading.py", line 123, in _table_last_update raise (e) File "/opt/conda/envs/epigraphhub/lib/python3.9/site-packages/epigraphhub/data/foph/loading.py", line 114, in _table_last_update df = pd.read_sql(f"select * from switzerland.foph_{table.lower()}_d;", engine) File "/opt/conda/envs/epigraphhub/lib/python3.9/site-packages/pandas/io/sql.py", line 592, in read_sql return pandas_sql.read_query( File "/opt/conda/envs/epigraphhub/lib/python3.9/site-packages/pandas/io/sql.py", line 1557, in read_query result = self.execute(*args) File "/opt/conda/envs/epigraphhub/lib/python3.9/site-packages/pandas/io/sql.py", line 1402, in execute return self.connectable.execution_options().execute(*args, **kwargs) File "", line 2, in execute File "/opt/conda/envs/epigraphhub/lib/python3.9/site-packages/sqlalchemy/util/deprecations.py", line 468, in warned return fn(*args, **kwargs) File "/opt/conda/envs/epigraphhub/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3257, in execute return connection.execute(statement, *multiparams, **params) File "/opt/conda/envs/epigraphhub/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1365, in execute return self._exec_driver_sql( File "/opt/conda/envs/epigraphhub/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1669, in _exec_driver_sql ret = self._execute_context( File "/opt/conda/envs/epigraphhub/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context self._handle_dbapi_exception( File "/opt/conda/envs/epigraphhub/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception util.raise_( File "/opt/conda/envs/epigraphhub/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_ raise exception File "/opt/conda/envs/epigraphhub/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context self.dialect.do_execute( File "/opt/conda/envs/epigraphhub/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.ProgrammingError: (psycopg2.errors.InsufficientPrivilege) permission denied for schema switzerland LINE 1: select * from switzerland.foph_casesvaccpersons_d;

PS: btw, in the worst case we could have it as a COPY statement in the dockerfile

I tried to run the sql file directly into the postgres container, all tables and schemas were created, I think only the grants didn't worked. The commands are ok tho, since after granting manually the dag works

xmnlab commented 1 year ago

sure, the logs are very similar

I mean the postgresql server log

xmnlab commented 1 year ago

but nvm .. we have a step that upload the logs as an artifact here https://github.com/thegraphnetwork/EpiGraphHub/suites/12074760051/artifacts/635510868

xmnlab commented 1 year ago 
postgres_1  | 2023-04-06 15:07:06.444 UTC [64] ERROR:  relation "foph_hospvaccpersons_pkey" already exists
postgres_1  | 2023-04-06 15:07:06.444 UTC [64] STATEMENT:  ALTER TABLE ONLY switzerland.foph_hospvaccpersons_d
postgres_1  |      ADD CONSTRAINT foph_hospvaccpersons_pkey PRIMARY KEY (id_);
postgres_1  | psql:/docker-entrypoint-initdb.d/12-epigraphhub.sql:18323: ERROR:  relation "foph_hospvaccpersons_pkey" already exists
postgres_1  | 
postgres_1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres_1  | 
postgres_1  | 2023-04-06 15:07:06.993 UTC [1] LOG:  starting PostgreSQL 14.7 (Debian 14.7-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
postgres_1  | 2023-04-06 15:07:06.993 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 25432
postgres_1  | 2023-04-06 15:07:06.993 UTC [1] LOG:  listening on IPv6 address "::", port 25432
postgres_1  | 2023-04-06 15:07:06.994 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.25432"
postgres_1  | 2023-04-06 15:07:06.998 UTC [26] LOG:  database system was interrupted; last known up at 2023-04-06 15:07:03 UTC
postgres_1  | 2023-04-06 15:07:07.039 UTC [26] LOG:  database system was not properly shut down; automatic recovery in progress
postgres_1  | 2023-04-06 15:07:07.042 UTC [26] LOG:  redo starts at 0/16FF730
postgres_1  | 2023-04-06 15:07:07.154 UTC [26] LOG:  invalid record length at 0/2E23090: wanted 24, got 0
postgres_1  | 2023-04-06 15:07:07.154 UTC [26] LOG:  redo done at 0/2E22F68 system usage: CPU: user: 0.06 s, system: 0.04 s, elapsed: 0.11 s
postgres_1  | 2023-04-06 15:07:07.280 UTC [1] LOG:  database system is ready to accept connections
postgres_1  | 2023-04-06 15:08:15.648 UTC [96] ERROR:  duplicate key value violates unique constraint "dag_pkey"
postgres_1  | 2023-04-06 15:08:15.648 UTC [96] DETAIL:  Key (dag_id)=(SINAN_DIFT) already exists.

@luabida, can you check that please?

luabida commented 1 year ago

I see, thank you for spotting it out

luabida commented 1 year ago

@xmnlab by any chance, do you why is CI accusing switzerland schema doesn't exist but the public.owid_covid does? https://github.com/thegraphnetwork/EpiGraphHub/actions/runs/4682606854/jobs/8296668536#step:14:4309 I'm getting out of ideas here, the schema creation happens even before the owid_covid table is created and it still says there is no schema switzerland.. https://github.com/thegraphnetwork/EpiGraphHub/blob/2d218052494b759d1e4c1dc2a31c579dc8008a95/containers/postgresql/sql/dev/02-epigraphhub.sql#L59

xmnlab commented 1 year ago

@luabida as your code is not about anything that would change the potgresql server or the dags, what I can suggest you for now is to move back containers/postgresql/sql/dev/02-epigraphhub.sql and we can merge this PR. and we can take a look into that later

luabida commented 1 year ago

@xmnlab selenium tests are passing.. ready to review & merge