Postgres Authentication issue while running the pipeline

[kadnan]

Hi

I am trying Mara very first time. Pretty good. From the demo, I am running the following code but getting the error

 Traceback (most recent call last):
  File "/Users/AdnanAhmad/Data/anaconda3/lib/python3.7/site-packages/mara_pipelines/execution.py", line 67, in run
    node_durations_and_run_times = node_cost.node_durations_and_run_times(pipeline)
  File "/Users/AdnanAhmad/Data/anaconda3/lib/python3.7/site-packages/mara_pipelines/logging/node_cost.py", line 22, in node_durations_and_run_times
    with mara_db.postgresql.postgres_cursor_context('mara') as cursor:
  File "/Users/AdnanAhmad/Data/anaconda3/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/Users/AdnanAhmad/Data/anaconda3/lib/python3.7/site-packages/mara_db/postgresql.py", line 19, in postgres_cursor_context
    host=db.host, port=db.port)  # type: psycopg2.extensions.connection
  File "/x/lib/python3.7/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: fe_sendauth: no password supplied

from mara_pipelines.commands.bash import RunBash
from mara_pipelines.pipelines import Pipeline, Task
from mara_pipelines.ui.cli import run_pipeline, run_interactively
from mara_pipelines.ui.cli import run_pipeline

pipeline = Pipeline(
    id='demo',
    description='A small pipeline that demonstrates the interplay between pipelines, tasks and commands')

pipeline.add(Task(id='ping_localhost', description='Pings localhost',
                  commands=[RunBash('ping -c 3 localhost')]))

sub_pipeline = Pipeline(id='sub_pipeline', description='Pings a number of hosts')

for host in ['google', 'amazon', 'facebook']:
    sub_pipeline.add(Task(id=f'ping_{host}', description=f'Pings {host}',
                          commands=[RunBash(f'ping -c 3 {host}.com')]))

sub_pipeline.add_dependency('ping_amazon', 'ping_facebook')
sub_pipeline.add(Task(id='ping_foo', description='Pings foo',
                      commands=[RunBash('ping foo')]), ['ping_amazon'])

pipeline.add(sub_pipeline, ['ping_localhost'])

pipeline.add(Task(id='sleep', description='Sleeps for 2 seconds',
                  commands=[RunBash('sleep 2')]), ['sub_pipeline'])

if __name__ == '__main__':
    run_pipeline(pipeline)

I had create a separate .py file in which I used the code to setup db

import mara_db.auto_migration
import mara_db.config
import mara_db.dbs

mara_db.config.databases \
    = lambda: {'mara': mara_db.dbs.PostgreSQLDB(
    host='localhost',
    user='postgres',
    password='postgres',
    database='example_etl_mara')}

mara_db.auto_migration.auto_discover_models_and_migrate()

It worked well and created db. Now it is not clear whether I have to run this code while running pipeline too?

Thanks

[martin-loetzsch]

Hi @kadnan,

the code that configures the databases needs to run before the pipeline in the same Pyhon process:

mara_db.config.databases \
    = lambda: {'mara': mara_db.dbs.PostgreSQLDB(
    host='localhost',
    user='postgres',
    password='postgres',
    database='example_etl_mara')}

The run_pipeline needs this configuration to connect to the right database. So I think the problem is to have two separate files / databases.

[hemanth7787]

@martin-loetzsch does the mara_db.auto_migration.auto_discover_models_and_migrate() supposed to create necessary tables ?

[leo-schick]

@kadnan The code part mara_db.auto_migration.auto_discover_models_and_migrate() as @hemanth7787 mentioned are required only once to make sure that the necessary mara tables are created in the database.

The code part mara_db.config.databases = lambda: ... is necessary on each pipeline run. This configures the mara database connection to mara.

I will close this issue now since it is quite old and seems to be outdated. Feel free to open it again.