Failure to query data from the core

CarolinaFernandez commented 2 years ago

Issue description

It is not possible to operate with data in the orchestrator (e.g., on processes). An Internal Server Error is returned, indicating the "processes" table is not available in the DB.

Steps to reproduce the issue

Instructions from the STF slides (pp. 14-17) were used (just changing the main.py name).

```
virtualenv venv --python=python3.9.5
```
```
source venv/bin/activate
```
```
(venv) pip install orchestrator-core
```

docker run --name postgres-orchestrator -p 5432:5432 -e POSTGRES_PASSWORD=nwa -d postgres

# set any password (here, "nwa")
docker exec -it postgres-orchestrator createuser -sP nwa -p 5432 -h 127.0.0.1 -U postgres

docker exec -it postgres-orchestrator createdb orchestrator-core -O nwa -p 5432 -h 127.0.0.1 -U postgres

docker run --name redis-orchestrator -p 6379:6379 -d redis

cat<<EOF>main_cli.py
from orchestrator import OrchestratorCore
from orchestrator.cli.main import app as core_cli
from orchestrator.settings import AppSettings

app = OrchestratorCore(base_settings=AppSettings())

if __name__ == "__main__":
   core_cli()
EOF

(venv) $ PYTHONPATH=./venv/lib/python3.9/site-packages python main_cli.py db init

(venv) $ PYTHONPATH=./venv/lib/python3.9/site-packages python main_cli.py db upgrade

(venv) $ PYTHONPATH=./venv/lib/python3.9/site-packages uvicorn --reload --host 127.0.0.1 --port 8080 main_cli:app

curl -X 'GET' http://127.0.0.1:8080/api/processes/ -H 'accept: application/json'

What's the expected result?

Not sure. Although I would guess an empty list of products at this stage.

What's the actual result?

When initialising the DB:

# (Command previously run in this environment, backing up and removing "migrations" folder)
mv migrations migrations.bak

# Initialising DB
(venv) $ PYTHONPATH=./venv/lib/python3.9/site-packages python main_cli.py db init
ujson module not found, using json
2022-03-29 20:09.13 [warning  ] WebSocketManager object configured, all methods referencing `websocket_manager` should work.
2022-03-29 20:09.13 [warning  ] DistLockManager object configured, all methods referencing `distlock_manager` should work.
2022-03-29 18:09:13 [warning  ] Database object configured, all methods referencing `db` should work. [orchestrator.db] 
2022-03-29 18:09:13 [info     ] Creating directory             [orchestrator.cli.database] directory=./surfnet/orchestrator-core/migrations
2022-03-29 18:09:13 [info     ] Creating directory             [orchestrator.cli.database] directory=./surfnet/orchestrator-core/migrations/versions
2022-03-29 18:09:13 [info     ] Creating directory             [orchestrator.cli.database] directory=./surfnet/orchestrator-core/migrations/versions/schema
2022-03-29 18:09:13 [info     ] Creating file                  [orchestrator.cli.database] file=./surfnet/orchestrator-core/migrations/env.py
2022-03-29 18:09:13 [info     ] Creating file                  [orchestrator.cli.database] file=./surfnet/orchestrator-core/migrations/script.py.mako
2022-03-29 18:09:13 [info     ] Creating file                  [orchestrator.cli.database] file=./surfnet/orchestrator-core/migrations/helpers.py
2022-03-29 18:09:13 [info     ] Skipping Alembic.ini file. It already exists [orchestrator.cli.database]

# Checking generated directory
(venv) $ tree migrations
migrations
├── env.py
├── helpers.py
├── script.py.mako
└── versions
    └── schema

When performing a DB upgrade:

(venv) $ PYTHONPATH=./venv/lib/python3.9/site-packages python main_cli.py db upgrade
(...)
2022-03-29 18:10:55 [info     ] Version Locations              [orchestrator.cli.database] locations=migrations/versions/general ./surfnet/orchestrator-core/orchestrator/migrations/versions/schema
Traceback (most recent call last):
  File "./surfnet/orchestrator-core/main_cli.py", line 10, in <module>
    core_cli()
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/typer/main.py", line 500, in wrapper
    return callback(**use_params)  # type: ignore
  File "./surfnet/orchestrator-core/orchestrator/cli/database.py", line 139, in upgrade
    command.upgrade(alembic_cfg(), revision)
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/alembic/command.py", line 277, in upgrade
    if ":" in revision:
TypeError: argument of type 'NoneType' is not iterable

When querying the API, it returns an Internal Server Error (see details from the API logs):

curl -X 'GET' http://127.0.0.1:8080/api/processes/ -H 'accept: application/json'

>> Details:
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
    self.dialect.do_execute(
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "processes" does not exist
LINE 2: FROM processes LEFT OUTER JOIN processes_subscriptions AS pr…
[SQL: SELECT processes.pid AS processes_pid, processes.workflow AS processes_workflow, processes.assignee AS processes_assignee, processes.last_status AS processes_last_status, processes.last_step AS processes_last_step, processes.started_at AS processes_started_at, processes.last_modified_at AS processes_last_modified_at, processes.failed_reason AS processes_failed_reason, processes.created_by AS processes_created_by, processes.is_task AS processes_is_task, products_1.product_id AS products_1_product_id, products_1.name AS products_1_name, products_1.description AS products_1_description, products_1.product_type AS products_1_product_type, products_1.tag AS products_1_tag, products_1.status AS products_1_status, products_1.created_at AS products_1_created_at, products_1.end_date AS products_1_end_date, subscriptions_1.subscription_id AS subscriptions_1_subscription_id, subscriptions_1.description AS subscriptions_1_description, subscriptions_1.status AS subscriptions_1_status, subscriptions_1.product_id AS subscriptions_1_product_id, subscriptions_1.customer_id AS subscriptions_1_customer_id, subscriptions_1.insync AS subscriptions_1_insync, subscriptions_1.start_date AS subscriptions_1_start_date, subscriptions_1.end_date AS subscriptions_1_end_date, subscriptions_1.note AS subscriptions_1_note, processes_subscriptions_1.id AS processes_subscriptions_1_id, processes_subscriptions_1.pid AS processes_subscriptions_1_pid, processes_subscriptions_1.subscription_id AS processes_subscriptions_1_subscription_id, processes_subscriptions_1.created_at AS processes_subscriptions_1_created_at, processes_subscriptions_1.workflow_target AS processes_subscriptions_1_workflow_target 
FROM processes LEFT OUTER JOIN processes_subscriptions AS processes_subscriptions_1 ON processes.pid = processes_subscriptions_1.pid LEFT OUTER JOIN subscriptions AS subscriptions_1 ON subscriptions_1.subscription_id = processes_subscriptions_1.subscription_id LEFT OUTER JOIN products AS products_1 ON products_1.product_id = subscriptions_1.product_id]

Database content shows just the DB, not the tables:

$ docker exec -it postgres-orchestrator /bin/bash
root@ac30141c3cc1:/# su postgres
postgres@ac30141c3cc1:/$ psql 
psql (14.2 (Debian 14.2-1.pgdg110+1))
Type "help" for help.

postgres=# \l
                                     List of databases
       Name        |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   
-------------------+----------+----------+------------+------------+-----------------------
 orchestrator-core | nwa      | UTF8     | en_US.utf8 | en_US.utf8 | 
 postgres          | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
 template0         | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
                   |          |          |            |            | postgres=CTc/postgres
 template1         | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
                   |          |          |            |            | postgres=CTc/postgres
(4 rows)

postgres=# \dt
Did not find any relations.

Used environment

OS: Ubuntu 20.04.4 LTS
Docker: 20.10.14, build a224086
Python: 3.8.10

Additional details / screenshot

N/A

acidjunk commented 2 years ago

Seems like the migrations are not executed. Could you try just starting the webserver: that would normally trigger the migrations. And you should be able to see that in the logs.

side note: python 3.8 or 3.9: the docker setup is 3.8, the rest of your description 3.9?

acidjunk commented 2 years ago

It seems that you are running the db init from inside the orchestrator-core folder. Did you start you own project and included orchestrator-core in the pip requirements or are you running the command in a clone of this repo?

CarolinaFernandez commented 2 years ago

Seems like the migrations are not executed. Could you try just starting the webserver: that would normally trigger the migrations. And you should be able to see that in the logs.

I understand that was step 11, prior to the cURL call. If that worked, should the Postgre DB have some relations defined at that stage? It does not have any.

side note: python 3.8 or 3.9: the docker setup is 3.8, the rest of your description 3.9?

Started with Python 3.8 as system default. After the issues to deploy, I installed python 3.9 in my system and set it as default via alternatives. However, it breaks some binaries, so I am back to python 3.8. In summary: currently it is python 3.8.10 system-wide and python 3.9.5 in venv (first time I try a venv with a different version than the one in the system, and it fails with missing libraries, i.e., pyscopg2). When trying the same version in venv, it complains about missing modules under sqlalchemy.exc

It seems that you are running the db init from inside the orchestrator-core folder. Did you start you own project and included orchestrator-core in the pip requirements or are you running the command in a clone of this repo?

The latter. When first starting with workflow-orchestrator, started with a clone of this project and used the development instructions (using flit, etc). In the last tests, used the pip requirement on orchestrator-core, as well as executing all commands under this same orchestrator-core repository folder.

I have tried now in a new folder. No repository of any kind there, just a new folder with the main_cli.py from above and a venv with the same version as Python system-wide, i.e., 3.8.10:

# Initialise DB
(venv) $ PYTHONPATH=. python main_cli.py db init
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
2022-03-30 15:45.08 [error    ] Could not get git commit hash  
Traceback (most recent call last):
  File "/home/carolina/.local/lib/python3.8/site-packages/orchestrator/version.py", line 39, in __getattr__
    return check_output(["/usr/bin/env", "git", "rev-parse", "HEAD"]).decode().strip()  # noqa: S603
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/env', 'git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
ujson module not found, using json
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
2022-03-30 15:45.09 [error    ] Could not get git commit hash  
Traceback (most recent call last):
  File "/home/carolina/.local/lib/python3.8/site-packages/orchestrator/version.py", line 39, in __getattr__
    return check_output(["/usr/bin/env", "git", "rev-parse", "HEAD"]).decode().strip()  # noqa: S603
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/env', 'git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
2022-03-30 13:45:09 [warning  ] Database object configured, all methods referencing `db` should work. [orchestrator.db] 
2022-03-30 13:45:09 [info     ] Creating directory             [orchestrator.cli.database] directory=./orchestrator-core__mine/migrations
2022-03-30 13:45:09 [info     ] Creating directory             [orchestrator.cli.database] directory=./orchestrator-core__mine/migrations/versions
2022-03-30 13:45:09 [info     ] Creating directory             [orchestrator.cli.database] directory=./orchestrator-core__mine/migrations/versions/schema
2022-03-30 13:45:09 [info     ] Creating file                  [orchestrator.cli.database] file=./orchestrator-core__mine/migrations/env.py
2022-03-30 13:45:09 [info     ] Creating file                  [orchestrator.cli.database] file=./orchestrator-core__mine/migrations/script.py.mako
2022-03-30 13:45:09 [info     ] Creating file                  [orchestrator.cli.database] file=./orchestrator-core__mine/migrations/helpers.py
2022-03-30 13:45:09 [info     ] Creating file                  [orchestrator.cli.database] file=./orchestrator-core__mine/alembic.ini

# Run webserver
(venv) $ PYTHONPATH=. uvicorn --reload --host 127.0.0.1 --port 8080 main_cli:app
INFO:     Uvicorn running on http://127.0.0.1:8080 (Press CTRL+C to quit)
INFO:     Started reloader process [39098] using watchgod
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
2022-03-30 15:46.45 [error    ] Could not get git commit hash  
Traceback (most recent call last):
  File "/home/carolina/.local/lib/python3.8/site-packages/orchestrator/version.py", line 39, in __getattr__
    return check_output(["/usr/bin/env", "git", "rev-parse", "HEAD"]).decode().strip()  # noqa: S603
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/env', 'git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
ujson module not found, using json
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
2022-03-30 15:46.46 [error    ] Could not get git commit hash  
Traceback (most recent call last):
  File "/home/carolina/.local/lib/python3.8/site-packages/orchestrator/version.py", line 39, in __getattr__
    return check_output(["/usr/bin/env", "git", "rev-parse", "HEAD"]).decode().strip()  # noqa: S603
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/env', 'git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
2022-03-30 13:46:46 [warning  ] Database object configured, all methods referencing `db` should work. [orchestrator.db] 
INFO:     Started server process [39100]
2022-03-30 13:46:46 [info     ] Started server process [39100] [uvicorn.error] 
INFO:     Waiting for application startup.
2022-03-30 13:46:46 [info     ] Waiting for application startup. [uvicorn.error] 
INFO:     Application startup complete.
2022-03-30 13:46:46 [info     ] Application startup complete.  [uvicorn.error]

No changes in the Postgre container w.r.t. the relations or the databases.

Also, when trying the upgrade command in this folder the same error from last time is produced:

(venv) $ PYTHONPATH=. python main_cli.py db upgrade
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
2022-03-30 15:48.22 [error    ] Could not get git commit hash  
Traceback (most recent call last):
  File "/home/carolina/.local/lib/python3.8/site-packages/orchestrator/version.py", line 39, in __getattr__
    return check_output(["/usr/bin/env", "git", "rev-parse", "HEAD"]).decode().strip()  # noqa: S603
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/env', 'git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
ujson module not found, using json
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
2022-03-30 15:48.23 [error    ] Could not get git commit hash  
Traceback (most recent call last):
  File "/home/carolina/.local/lib/python3.8/site-packages/orchestrator/version.py", line 39, in __getattr__
    return check_output(["/usr/bin/env", "git", "rev-parse", "HEAD"]).decode().strip()  # noqa: S603
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/env', 'git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
2022-03-30 13:48:23 [warning  ] Database object configured, all methods referencing `db` should work. [orchestrator.db] 
2022-03-30 13:48:23 [info     ] Version Locations              [orchestrator.cli.database] locations=migrations/versions/general /home/carolina/.local/lib/python3.8/site-packages/orchestrator/migrations/versions/schema
Traceback (most recent call last):
  File "main_cli.py", line 10, in <module>
    core_cli()
  File "/home/carolina/.local/lib/python3.8/site-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "/home/carolina/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/carolina/.local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/carolina/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/carolina/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/carolina/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/carolina/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/carolina/.local/lib/python3.8/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/carolina/.local/lib/python3.8/site-packages/orchestrator/cli/database.py", line 126, in upgrade
    command.upgrade(alembic_cfg(), revision)
  File "/home/carolina/.local/lib/python3.8/site-packages/alembic/command.py", line 277, in upgrade
    if ":" in revision:
TypeError: argument of type 'NoneType' is not iterable

Finally, the cURL command returns the same issue regarding the missing "processes" relation.

PS: I recreated the postgres-orchestrator Docker instance from scratch. The "orchestrator-core" database was not created this time after executing the "db init" command in the new folder, outside the orchestrator-core repository.

acidjunk commented 2 years ago

I've discussed this during our standup. Tomorrow I will have some time to start a small example project; so I can try to reproduce your problem.

acidjunk commented 2 years ago

I created an repo that shows how you can use the orchestrator from inside your own project:

https://github.com/acidjunk/orchestrator-demo

When you follow the README, setup a postgres user orchestrator-demo with password orchestrator-demo and installed all python deps + loaded the needed ENV vars, this should yield you a working DB:

Note: redis should be running...

dropdb orchestrator-demo
createdb orchestrator-demo
./start.sh

You can see that it runs the migrations from Core 0.4.0-RC6.

I also added an initial migration in the project itself to demonstrate how you can add your own tables.

Tested API endpoint (returns empty list, which is ok):

Schermafbeelding 2022-04-02 om 01 48 50

acidjunk commented 2 years ago

whilst working in this I think I encountered the same bug: I needed to create an init migration for my own orchestrator project so I did what the README mentioned:

PYTHONPATH=. python main.py db init -> that created a migrations folder with some files.

Then I wanted to create my own migration for the UserPreference model in my own project but the README doesn't cover it. I tried a couple of approaches but when a migration was created it would be inside the venvs site_packages folder instead of migrations/versions.

`PYTHONPATH=. python main.py db revision --autogenerate -m "New schema" --head=schema@head

(I know that the --head parameter should probably contain something else; but not sure what....)

So I copied it myself to the correct place, to get it working.

Curious if this also happened with your setup.

acidjunk commented 2 years ago

@CarolinaFernandez did you manage to get your own project working?

CarolinaFernandez commented 2 years ago

Hi Rene,

I tried following the instructions from the new repository you created (https://github.com/acidjunk/orchestrator-demo.git) and then, the instructions from your message of 2nd April 2022. To that end, this time I ditched Docker and created a new VM in Vagrant (with Virtualbox), using Ubuntu 22.04 LTS, with Python 3.10.4, PostgreSQL 14 and Redis 5:6.0.16-1ubuntu1.

Summing up, I am now able to query the API (e.g., http://localhost:8080/api/products) without errors, so that it returns the expected empty list and where the server returns a 307 (temporary redirect). However, I can see no migrations (I guess it should be the file "migrations/versions/general/2022-04-02_022505714cf8_add_user_preference.py"?) when executing the command PYTHONPATH=. python main.py db init (I get "OSError: Directory migrations already exists and is not empty" and nothing new is prompted at the server side), so no chance of defining tables atm. I am also trying to setup the Vagrant environment properly so to access the Swagger UI from outside and try to force the creation of some resources to see which errors result from the persisting attempt.

Commands used below:

# Install Python3.10-dev (for venv)
sudo apt install -y python3.10-venv
# Install PostgreSQL
sudo apt install -y postgresql

#
# 1 . Perform steps from README.md
#

# Download specific repository
mkdir orchestrator-demo
cd orchestrator-demo
# git clone git@github.com:acidjunk/orchestrator-demo.git
git clone https://github.com/acidjunk/orchestrator-demo.git
cd orchestrator-demo
# Create venv and install all dependencies
python3 -m venv .venv
source .venv/bin/activate
pip install -r ./requirements/all.txt

# Create PostgreSQL user
# IMPORTANT: add "postgresql" to the "vagrant" group in /etc/group,
# so that PostgreSQL commands can access the provided migrations
# and also sudo as user before running these commands
sudo -u postgres -i
createuser -s -P orchestrator-demo
createdb orchestrator-demo -O orchestrator-demo
createdb orchestrator-demo-test -O orchestrator-demo
exit

sudo systemctl restart postgresql
sudo systemctl status postgresql
# May 15 22:50:49 ubuntu-jammy systemd[1]: Starting PostgreSQL RDBMS...
# May 15 22:50:49 ubuntu-jammy systemd[1]: Finished PostgreSQL RDBMS.

# Install Redis
sudo apt install -y redis-server
sudo systemctl status redis
# ● redis-server.service - Advanced key-value store
#      Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor preset: enabled)
#      Active: active (running) since Sun 2022-05-15 22:55:27 UTC; 6s ago
#        Docs: http://redis.io/documentation,
#              man:redis-server(1)
#    Main PID: 6299 (redis-server)
#      Status: "Ready to accept connections"
#       Tasks: 5 (limit: 4770)
#      Memory: 2.6M
#         CPU: 70ms
#      CGroup: /system.slice/redis-server.service
#              └─6299 "/usr/bin/redis-server 127.0.0.1:6379" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "">
# 
# May 15 22:55:27 ubuntu-jammy systemd[1]: Starting Advanced key-value store...
# May 15 22:55:27 ubuntu-jammy systemd[1]: Started Advanced key-value store.

# Run in development mode
cat<<EOF>>.env
export HTTP_PORT=8080
export MAIL_PORT=1025
export NSO_ENABLED=True
export OAUTH2_ACTIVE=False
export STACK_LIMIT=""
export SERVER_THREADED=1
export TESTING=False
export DATABASE_URI="postgresql://orchestrator-demo:orchestrator-demo@localhost/orchestrator-demo"
export DATABASE_URI_TEST="postgresql://orchestrator-demo:orchestrator-demo@localhost/orchestrator-demo-test"
export WEBSOCKET_BROADCASTER_URL="redis://localhost:6379"

# Populator config
export POPULATOR_ORGANISATION="GEANT"
export POPULATOR_CONTACT_NAME="Jane Doe"
export POPULATOR_CONTACT_EMAIL="jane.doe@nonexisting.net"
export POPULATOR_CONTACT_PHONE="0099700900500" # optional
EOF

#
# 2. Perform steps from the GitHub issue tracker
#

# IMPORTANT: add "postgresql" to the "vagrant" group in /etc/group,
# so that PostgreSQL commands can access the provided migrations
# and also sudo as user before running these commands
sudo -u postgres -i
dropdb orchestrator-demo
createdb orchestrator-demo
exit

## You can also ensure that you loaded the needed env vars and run a hot reloading webserver
DATABASE_URI=postgresql://orchestrator-demo:orchestrator-demo@localhost/orchestrator-demo PYTHONPATH=. uvicorn main:app --reload --port 8080
# INFO:     Will watch for changes in these directories: ['/home/vagrant/orchestrator-demo/orchestrator-demo']
# INFO:     Uvicorn running on http://127.0.0.1:8080 (Press CTRL+C to quit)
# INFO:     Started reloader process [5916] using watchgod
# ujson module not found, using json
# msgpack not installed, MsgPackSerializer unavailable
# 2022-05-15 23:22.30 [debug    ] Starting the app with the following settings app_settings=TESTING=True SESSION_SECRET='fgdJSOCPKKiJqWuZ' CORS_ORIGINS='*' CORS_ALLOW_METHODS=['GET', 'PUT', 'PATCH', 'POST', 'DELETE', 'OPTIONS', 'HEAD'] CORS_ALLOW_HEADERS=['If-None-Match', 'Authorization', 'If-Match', 'Content-Type'] CORS_EXPOSE_HEADERS=['Cache-Control', 'Content-Language', 'Content-Length', 'Content-Type', 'Expires', 'Last-Modified', 'Pragma', 'Content-Range', 'ETag'] ENVIRONMENT='local' WORKFLOWS_SWAGGER_HOST='localhost' WORKFLOWS_GUI_URI='http://localhost:3000' DATABASE_URI='postgresql://orchestrator-demo:orchestrator-demo@localhost/orchestrator-demo' MAX_WORKERS=5 MAIL_SERVER='localhost' MAIL_PORT=25 MAIL_STARTTLS=False CACHE_HOST='127.0.0.1' CACHE_PORT=6379 ENABLE_DISTLOCK_MANAGER=True DISTLOCK_BACKEND='redis' CC_NOC=0 SERVICE_NAME='orchestrator-core' LOGGING_HOST='localhost' LOG_LEVEL='DEBUG' SLACK_ENGINE_SETTINGS_HOOK_ENABLED=False SLACK_ENGINE_SETTINGS_HOOK_URL='' TRACING_ENABLED=False TRANSLATIONS_DIR=PosixPath('/home/vagrant/orchestrator-demo/orchestrator-demo/company/translations') WEBSOCKET_BROADCASTER_URL='redis://127.0.0.1:6379' ENABLE_WEBSOCKETS=True DISABLE_INSYNC_CHECK=False
# 2022-05-15 23:22.30 [warning  ] WebSocketManager object configured, all methods referencing `websocket_manager` should work.
# 2022-05-15 23:22.30 [warning  ] DistLockManager object configured, all methods referencing `distlock_manager` should work.
# 2022-05-15 23:22:30 [warning  ] Database object configured, all methods referencing `db` should work. [orchestrator.db]
# INFO:     Application startup complete.
# 2022-05-15 23:22:30 [info     ] Application startup complete.  [uvicorn.error]
# INFO:     Started server process [6713]
# 2022-05-15 23:22:30 [info     ] Started server process [6713]  [uvicorn.error]
# INFO:     Waiting for application startup.
# 2022-05-15 23:22:30 [info     ] Waiting for application startup. [uvicorn.error]
# 2022-05-15 23:22:30 [info     ] Connecting to redis            [asyncio_redis]
# 2022-05-15 23:22:30 [info     ] Redis connection made          [asyncio_redis]
# 2022-05-15 23:22:30 [info     ] Connecting to redis            [asyncio_redis]
# 2022-05-15 23:22:30 [info     ] Redis connection made          [asyncio_redis]
# INFO:     Application startup complete.

# Perform some cURL tests
curl http://localhost:8080/api/products
curl http://localhost:8080/api/products2
# {"detail":"Not Found"}

# INFO:     127.0.0.1:57672 - "GET /api/products HTTP/1.1" 307 Temporary Redirect
# INFO:     127.0.0.1:57674 - "GET /api/products2 HTTP/1.1" 404 Not Found

pboers1988 commented 2 years ago

Hi @CarolinaFernandez If I understand you correctly at the moment the tables aren't created at the moment? What is the output of the following call. I expect a stacktrace when the database is not initialised:

curl http://localhost:8080/api/products/

Note the trailing slash. As you can see in the logs the first call returned a Temp Redirect that was not followed. (You need to use curl -L) to follow redirects on the CLI.

Another question I have is as follows:

What is the out put of the following command:

PYTHONPATH=. python main.py db upgrade heads

Does that yield any results. If that commands runs correctly I curious to know what the results of products curl call is.

CarolinaFernandez commented 2 years ago

Hi @pboers1988. Indeed, tables are not being created in the DB.

1. Call to products endpoint

Reduced log output for curl -L http://localhost:8080/api/products/:

(.venv) vagrant@ubuntu-jammy:~/orchestrator-demo/orchestrator-demo$ DATABASE_URI=postgresql://orchestrator-demo:orchestrator-demo@localhost/orchestrator-demo PYTHONPATH=. uvicorn main:app --reload --port 8080
INFO:     Will watch for changes in these directories: ['/home/vagrant/orchestrator-demo/orchestrator-demo']
INFO:     Uvicorn running on http://127.0.0.1:8080 (Press CTRL+C to quit)
INFO:     Started reloader process [4001] using watchgod
ujson module not found, using json
msgpack not installed, MsgPackSerializer unavailable
2022-05-19 18:32.04 [debug    ] Starting the app with the following settings app_settings=TESTING=True SESSION_SECRET='iVXSBPhlOIpZcoln' CORS_ORIGINS='*' CORS_ALLOW_METHODS=['GET', 'PUT', 'PATCH', 'POST', 'DELETE', 'OPTIONS', 'HEAD'] CORS_ALLOW_HEADERS=['If-None-Match', 'Authorization', 'If-Match', 'Content-Type'] CORS_EXPOSE_HEADERS=['Cache-Control', 'Content-Language', 'Content-Length', 'Content-Type', 'Expires', 'Last-Modified', 'Pragma', 'Content-Range', 'ETag'] ENVIRONMENT='local' WORKFLOWS_SWAGGER_HOST='localhost' WORKFLOWS_GUI_URI='http://localhost:3000' DATABASE_URI='postgresql://orchestrator-demo:orchestrator-demo@localhost/orchestrator-demo' MAX_WORKERS=5 MAIL_SERVER='localhost' MAIL_PORT=25 MAIL_STARTTLS=False CACHE_HOST='127.0.0.1' CACHE_PORT=6379 ENABLE_DISTLOCK_MANAGER=True DISTLOCK_BACKEND='redis' CC_NOC=0 SERVICE_NAME='orchestrator-core' LOGGING_HOST='localhost' LOG_LEVEL='DEBUG' SLACK_ENGINE_SETTINGS_HOOK_ENABLED=False SLACK_ENGINE_SETTINGS_HOOK_URL='' TRACING_ENABLED=False TRANSLATIONS_DIR=PosixPath('/home/vagrant/orchestrator-demo/orchestrator-demo/company/translations') WEBSOCKET_BROADCASTER_URL='redis://127.0.0.1:6379' ENABLE_WEBSOCKETS=True DISABLE_INSYNC_CHECK=False
2022-05-19 18:32.04 [warning  ] WebSocketManager object configured, all methods referencing `websocket_manager` should work.
2022-05-19 18:32.04 [warning  ] DistLockManager object configured, all methods referencing `distlock_manager` should work.
2022-05-19 18:32:04 [warning  ] Database object configured, all methods referencing `db` should work. [orchestrator.db] 
INFO:     Started server process [4003]
2022-05-19 18:32:04 [info     ] Started server process [4003]  [uvicorn.error] 
INFO:     Waiting for application startup.
2022-05-19 18:32:04 [info     ] Waiting for application startup. [uvicorn.error] 
2022-05-19 18:32:04 [info     ] Connecting to redis            [asyncio_redis] 
2022-05-19 18:32:04 [info     ] Redis connection made          [asyncio_redis] 
2022-05-19 18:32:04 [info     ] Connecting to redis            [asyncio_redis] 
2022-05-19 18:32:04 [info     ] Redis connection made          [asyncio_redis] 
INFO:     Application startup complete.
2022-05-19 18:32:04 [info     ] Application startup complete.  [uvicorn.error] 
INFO:     127.0.0.1:55202 - "GET /api/products/ HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
    self.dialect.do_execute(
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "products" does not exist
LINE 2: FROM products
             ^
(...)

[SQL: SELECT products.product_id AS products_product_id, products.name AS products_name, products.description AS products_description, products.product_type AS products_product_type, products.tag AS products_tag, products.status AS products_status, products.created_at AS products_created_at, products.end_date AS products_end_date 
FROM products]
(Background on this error at: https://sqlalche.me/e/14/f405)

Output from Postgres (NOTE: I realised later that this connected to some database, I guess "postgres" that is not the orchestrator one. I cannot ensure the tables were not created at this moment):

vagrant@ubuntu-jammy:~$ sudo -u postgres -i
postgres@ubuntu-jammy:~$ psql 
psql (14.2 (Ubuntu 14.2-1ubuntu1))
Type "help" for help.

postgres=# \dt
Did not find any relations.
postgres=# \l
                                         List of databases
          Name          |       Owner       | Encoding | Collate |  Ctype  |   Access privileges   
------------------------+-------------------+----------+---------+---------+-----------------------
 orchestrator-demo      | postgres          | UTF8     | C.UTF-8 | C.UTF-8 | 
 orchestrator-demo-test | orchestrator-demo | UTF8     | C.UTF-8 | C.UTF-8 | 
 postgres               | postgres          | UTF8     | C.UTF-8 | C.UTF-8 | 
 template0              | postgres          | UTF8     | C.UTF-8 | C.UTF-8 | =c/postgres          +
                        |                   |          |         |         | postgres=CTc/postgres
 template1              | postgres          | UTF8     | C.UTF-8 | C.UTF-8 | =c/postgres          +
                        |                   |          |         |         | postgres=CTc/postgres
(5 rows)
postgres=# \du
                                       List of roles
     Role name     |                         Attributes                         | Member of 
-------------------+------------------------------------------------------------+-----------
 orchestrator-demo | Superuser, Create role, Create DB                          | {}
 postgres          | Superuser, Create role, Create DB, Replication, Bypass RLS | {}

2. Upgrade heads on DB

The command fails. Logs below:

vagrant@ubuntu-jammy:~/orchestrator-demo/orchestrator-demo$ source .venv/bin/activate
(.venv) vagrant@ubuntu-jammy:~/orchestrator-demo/orchestrator-demo$ PYTHONPATH=. python main.py db upgrade heads
ujson module not found, using json
msgpack not installed, MsgPackSerializer unavailable
2022-05-19 18:39.19 [debug    ] Starting the app with the following settings app_settings=TESTING=True SESSION_SECRET='HFAsBeKyOkZBdOAy' CORS_ORIGINS='*' CORS_ALLOW_METHODS=['GET', 'PUT', 'PATCH', 'POST', 'DELETE', 'OPTIONS', 'HEAD'] CORS_ALLOW_HEADERS=['If-None-Match', 'Authorization', 'If-Match', 'Content-Type'] CORS_EXPOSE_HEADERS=['Cache-Control', 'Content-Language', 'Content-Length', 'Content-Type', 'Expires', 'Last-Modified', 'Pragma', 'Content-Range', 'ETag'] ENVIRONMENT='local' WORKFLOWS_SWAGGER_HOST='localhost' WORKFLOWS_GUI_URI='http://localhost:3000' DATABASE_URI='postgresql://nwa:nwa@localhost/orchestrator-core' MAX_WORKERS=5 MAIL_SERVER='localhost' MAIL_PORT=25 MAIL_STARTTLS=False CACHE_HOST='127.0.0.1' CACHE_PORT=6379 ENABLE_DISTLOCK_MANAGER=True DISTLOCK_BACKEND='redis' CC_NOC=0 SERVICE_NAME='orchestrator-core' LOGGING_HOST='localhost' LOG_LEVEL='DEBUG' SLACK_ENGINE_SETTINGS_HOOK_ENABLED=False SLACK_ENGINE_SETTINGS_HOOK_URL='' TRACING_ENABLED=False TRANSLATIONS_DIR=PosixPath('/home/vagrant/orchestrator-demo/orchestrator-demo/company/translations') WEBSOCKET_BROADCASTER_URL='redis://127.0.0.1:6379' ENABLE_WEBSOCKETS=True DISABLE_INSYNC_CHECK=False
2022-05-19 18:39.19 [warning  ] WebSocketManager object configured, all methods referencing `websocket_manager` should work.
2022-05-19 18:39.19 [warning  ] DistLockManager object configured, all methods referencing `distlock_manager` should work.
2022-05-19 18:39:19 [warning  ] Database object configured, all methods referencing `db` should work. [orchestrator.db] 
2022-05-19 18:39:19 [info     ] Version Locations              [orchestrator.cli.database] locations=/home/vagrant/orchestrator-demo/orchestrator-demo/migrations/versions/general /home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/orchestrator/migrations/versions/schema
Traceback (most recent call last):
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3250, in _wrap_pool_connect
    return fn()
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 310, in connect
    return _ConnectionFairy._checkout(self)
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 868, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 476, in checkout
    rec = pool._do_get()
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 256, in _do_get
    return self._create_connection()
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 256, in _create_connection
    return _ConnectionRecord(self)
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 371, in __init__
    self.__connect()
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 665, in __connect
    with util.safe_reraise():
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 590, in connect
    return dialect.connect(*cargs, **cparams)
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 597, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL:  password authentication failed for user "nwa"
FATAL:  password authentication failed for user "nwa"
(...)

(Background on this error at: https://sqlalche.me/e/14/e3q8)

Although the list of databases above seems to show that "postgres" is the owner, I get an error regarding password auth error on user "nwa". In this environment (a clean VM) I have not configured anything with that user. Unsure on why this is happening, as "DATABASE_URI" is passed explicitly when running uvicorn (although the connection string to the DB might suggest that it is connecting to "orchestrator-demo" database with the same user and password, whereas the database is owned by postgres).

Also, when grepping for the "nwa" term in the orchestrator-demo repo I see the METADATA file of the orchestrator_core package details the following commands when installed via pip : createuser -sP nwa, createdb orchestrator-core -O nwa, createuser -sP nwa, createdb orchestrator-core-test -O nwa. Just noticed this (nwa-stdlib) is another automation package from the workfloworchestrator repo and installed as dependency of the core.

Update 1: the command for the "upgrade heads" did not force the proper connection string to the DB, and was taking "nwa" (DATABASE_URI='postgresql://nwa:nwa@localhost/orchestrator-core') by default, thus failing.

New log output:

(.venv) vagrant@ubuntu-jammy:~/orchestrator-demo/orchestrator-demo$ DATABASE_URI=postgresql://orchestrator-demo:orchestrator-demo@localhost/orchestrator-demo PYTHONPATH=. python main.py db upgrade heads
ujson module not found, using json
msgpack not installed, MsgPackSerializer unavailable
2022-05-19 19:17.42 [debug    ] Starting the app with the following settings app_settings=TESTING=True SESSION_SECRET='CFaDpwmdRJlchkMv' CORS_ORIGINS='*' CORS_ALLOW_METHODS=['GET', 'PUT', 'PATCH', 'POST', 'DELETE', 'OPTIONS', 'HEAD'] CORS_ALLOW_HEADERS=['If-None-Match', 'Authorization', 'If-Match', 'Content-Type'] CORS_EXPOSE_HEADERS=['Cache-Control', 'Content-Language', 'Content-Length', 'Content-Type', 'Expires', 'Last-Modified', 'Pragma', 'Content-Range', 'ETag'] ENVIRONMENT='local' WORKFLOWS_SWAGGER_HOST='localhost' WORKFLOWS_GUI_URI='http://localhost:3000' DATABASE_URI='postgresql://orchestrator-demo:orchestrator-demo@localhost/orchestrator-demo' MAX_WORKERS=5 MAIL_SERVER='localhost' MAIL_PORT=25 MAIL_STARTTLS=False CACHE_HOST='127.0.0.1' CACHE_PORT=6379 ENABLE_DISTLOCK_MANAGER=True DISTLOCK_BACKEND='redis' CC_NOC=0 SERVICE_NAME='orchestrator-core' LOGGING_HOST='localhost' LOG_LEVEL='DEBUG' SLACK_ENGINE_SETTINGS_HOOK_ENABLED=False SLACK_ENGINE_SETTINGS_HOOK_URL='' TRACING_ENABLED=False TRANSLATIONS_DIR=PosixPath('/home/vagrant/orchestrator-demo/orchestrator-demo/company/translations') WEBSOCKET_BROADCASTER_URL='redis://127.0.0.1:6379' ENABLE_WEBSOCKETS=True DISABLE_INSYNC_CHECK=False
2022-05-19 19:17.42 [warning  ] WebSocketManager object configured, all methods referencing `websocket_manager` should work.
2022-05-19 19:17.42 [warning  ] DistLockManager object configured, all methods referencing `distlock_manager` should work.
2022-05-19 19:17:42 [warning  ] Database object configured, all methods referencing `db` should work. [orchestrator.db] 
2022-05-19 19:17:42 [info     ] Version Locations              [orchestrator.cli.database] locations=/home/vagrant/orchestrator-demo/orchestrator-demo/migrations/versions/general /home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/orchestrator/migrations/versions/schema
2022-05-19 19:17:42 [info     ] Context impl PostgresqlImpl.   [alembic.runtime.migration] 
2022-05-19 19:17:42 [info     ] Will assume transactional DDL. [alembic.runtime.migration] 
2022-05-19 19:17:42 [info     ] Running upgrade  -> c112305b07d3, Initial schema migration. [alembic.runtime.migration] 
2022-05-19 19:17:42 [debug    ] new branch insert c112305b07d3 [alembic.runtime.migration] 
2022-05-19 19:17:42 [info     ] Running upgrade c112305b07d3 -> a76b9185b334, Add generic workflows to core. [alembic.runtime.migration] 
2022-05-19 19:17:42 [debug    ] update c112305b07d3 to a76b9185b334 [alembic.runtime.migration] 
2022-05-19 19:17:42 [info     ] Running upgrade a76b9185b334 -> 3323bcb934e7, Fix tsv triggers. [alembic.runtime.migration] 
2022-05-19 19:17:42 [debug    ] update a76b9185b334 to 3323bcb934e7 [alembic.runtime.migration] 
2022-05-19 19:17:42 [info     ] Running upgrade 3323bcb934e7 -> 3c8b9185c221, Add task_validate_products. [alembic.runtime.migration] 
2022-05-19 19:17:42 [debug    ] update 3323bcb934e7 to 3c8b9185c221 [alembic.runtime.migration] 
2022-05-19 19:17:42 [info     ] Running upgrade 3c8b9185c221 -> 6896a54e9483, Add product_block_relations. [alembic.runtime.migration] 
2022-05-19 19:17:42 [debug    ] update 3c8b9185c221 to 6896a54e9483 [alembic.runtime.migration] 
2022-05-19 19:17:42 [info     ] Running upgrade 6896a54e9483 -> 19cdd3ab86f6, fix_parse_websearch. [alembic.runtime.migration] 
2022-05-19 19:17:42 [debug    ] update 6896a54e9483 to 19cdd3ab86f6 [alembic.runtime.migration] 
2022-05-19 19:17:42 [info     ] Running upgrade 19cdd3ab86f6 -> bed6bc0b197a, rename_parent_and_child_block_relations. [alembic.runtime.migration] 
2022-05-19 19:17:42 [debug    ] update 19cdd3ab86f6 to bed6bc0b197a [alembic.runtime.migration] 
2022-05-19 19:17:42 [info     ] Running upgrade bed6bc0b197a -> 022505714cf8, Add user preference. [alembic.runtime.migration] 
2022-05-19 19:17:42 [debug    ] update bed6bc0b197a to 022505714cf8 [alembic.runtime.migration]

New call to products:

(.venv) vagrant@ubuntu-jammy:~/orchestrator-demo/orchestrator-demo$ curl -L http://localhost:8080/api/products/
[]

Now it is indeed able to query the service w/o server errors.

Update 2: also realised that my last dt was carried out on an unknown database. If forcing the change to the orchestrator-demo DB:

postgres=# \c orchestrator-demo
You are now connected to database "orchestrator-demo" as user "postgres".
orchestrator-demo=# \dt
                            List of relations
 Schema |                Name                | Type  |       Owner       
--------+------------------------------------+-------+-------------------
 public | alembic_version                    | table | orchestrator-demo
 public | engine_settings                    | table | orchestrator-demo
 public | fixed_inputs                       | table | orchestrator-demo
 public | process_steps                      | table | orchestrator-demo
 public | processes                          | table | orchestrator-demo
 public | processes_subscriptions            | table | orchestrator-demo
 public | product_block_relations            | table | orchestrator-demo
 public | product_block_resource_types       | table | orchestrator-demo
 public | product_blocks                     | table | orchestrator-demo
 public | product_product_blocks             | table | orchestrator-demo
 public | products                           | table | orchestrator-demo
 public | products_workflows                 | table | orchestrator-demo
 public | resource_types                     | table | orchestrator-demo
 public | subscription_customer_descriptions | table | orchestrator-demo
 public | subscription_instance_relations    | table | orchestrator-demo
 public | subscription_instance_values       | table | orchestrator-demo
 public | subscription_instances             | table | orchestrator-demo
 public | subscriptions                      | table | orchestrator-demo
 public | user_preference                    | table | orchestrator-demo
 public | workflows                          | table | orchestrator-demo
(20 rows)

As a final comment/question (not to be answered if the assumption is correct), I see the "user_preference" table has a match under "migrations/versions/general/2022-04-02_022505714cf8_add_user_preference.py" . I assume that migrations are then used either to create new custom tables or to insert entries in the already existing/default tables, like in this example).

Given that there are no longer server errors and the products and workflows show now an empty and a filled list, respectively; it is time to close the ticket and find out how to populate other structures. Thanks all who worked on this and replied here, and special thanks on the updated instructions and repo.

acidjunk commented 2 years ago

🎉

Migrations can indeed be used to create tables AND to insert/populate data. We use them also to add a new workflow.

Reading the last comments I think it would be nice to provide a docker/vagrant based environment to make it easier; to play with the demo project in a "known environment". 10 million ways to run python.

There seem to be problems regarding setting up stuff to correctly run the initial migration...

Thanks for reporting back!

CarolinaFernandez commented 2 years ago

Thanks for the clarification on the migrations and the support before, @acidjunk ! There is much material to read and test when continuing with this analysis in the meantime.

Reading the last comments I think it would be nice to provide a docker/vagrant based environment to make it easier; to play with the demo project in a "known environment". 10 million ways to run python.

Definitely seconding this. I think that a Docker environment would be ideal, otherwise a Vagrant one. In any case, whatever allows replicability and provides a base for developers to (hopefully) also packet their extended logic. As these are both self-contained I foresee no help needed there for testing, but otherwise drop me a line.

workfloworchestrator / orchestrator-core