Bad terminal handling of `airflow db shell` command with Postgres DB

[blag]

Apache Airflow version

main (development)

What happened

This happens in both iTerm 2 and the built-in Terminal.app in macOS when running ~breeze start-airflow~. airflow db shell when using postgres as a backend.

Within psql (run via airflow db shell):

• Scrollback editing (eg: hitting up arrow to edit a previous query) is messed up for multi-line queries. I don't have a better description. I recorded a video of what happens.
• Some things require an additional keypress (eg: quitting the pager)
  • Some things, like ctrl+P to scrollback needing an additional keypress happen in a normal Docker environment. This is different and distinct from that.
• SIGINT isn't registered when trying to cancel editing a query and reset the SQL prompt

I suspect that there's a lot of interaction happening between iTerm/Terminal.app, ZSH, Python/Breeze, Bash, Docker, tmux, Python/Airflow CLI, and psql, and some keypresses are being filtered or queued along the way. So there are multiple fun rabbit holes to chase down to solve this.

What you think should happen instead

• Scrollback editing should work. Your text cursor should be in the correct place and the text should be rendered correctly.
• An additional keypress to exit the pager should not be necessary
• SIGINT should cancel editing the query and reset the edit line (eg: it should return to an empty SQL prompt: airflow=# instead of airflow-#)

How to reproduce

• Scrollback editing is messed up - indentation and text wrapping are off/wrong

  • This happens even when I don't adjust the tmux pane sizes at all
  • Example:
  • Run breeze start-airflow in a terminal
  • Run docker exec -it $CONTAINER_ID /bin/bash with the breeze container that is running
  • Run airflow db shell within the container

  • Then this happens (note the up arrow keypresses aren't called out visually):

    https://user-images.githubusercontent.com/597113/188750191-2b3e8ef3-ee89-4b2e-872d-1451807bdbc7.mov

• Some things require an additional keypress Example:

  • Run a SQL query:

    airflow=# SELECT * FROM dag_schedule_dataset_reference

  • it displays in the pager

    dataset_id |        dag_id         |          created_at           |          updated_at
    ------------+-----------------------+-------------------------------+-------------------------------
            3 | consumer_dag_2        | 2022-09-06 20:49:23.719275+00 | 2022-09-06 20:49:23.719295+00
            2 | consumer_dag_2        | 2022-09-06 20:49:23.7193+00   | 2022-09-06 20:49:23.719302+00
            4 | taskflow_consumer_dag | 2022-09-06 20:49:23.719305+00 | 2022-09-06 20:49:23.719308+00
            5 | taskflow_consumer_dag | 2022-09-06 20:49:23.719311+00 | 2022-09-06 20:49:23.719313+00
            6 | taskflow_consumer_dag | 2022-09-06 20:49:23.719316+00 | 2022-09-06 20:49:23.719319+00
            1 | consumer_dag_1        | 2022-09-06 20:49:23.719322+00 | 2022-09-06 20:49:23.719324+00
    (6 rows)
    
    (END)

  • Usually you just hit q and it exits immediately, but the pager sticks around:

    dataset_id |        dag_id         |          created_at           |          updated_at
    ------------+-----------------------+-------------------------------+-------------------------------
            3 | consumer_dag_2        | 2022-09-06 20:49:23.719275+00 | 2022-09-06 20:49:23.719295+00
            2 | consumer_dag_2        | 2022-09-06 20:49:23.7193+00   | 2022-09-06 20:49:23.719302+00
            4 | taskflow_consumer_dag | 2022-09-06 20:49:23.719305+00 | 2022-09-06 20:49:23.719308+00
            5 | taskflow_consumer_dag | 2022-09-06 20:49:23.719311+00 | 2022-09-06 20:49:23.719313+00
            6 | taskflow_consumer_dag | 2022-09-06 20:49:23.719316+00 | 2022-09-06 20:49:23.719319+00
            1 | consumer_dag_1        | 2022-09-06 20:49:23.719322+00 | 2022-09-06 20:49:23.719324+00
    (6 rows)
    
    (END)

  • The q keypress is registered upon your next keypress, whatever that is (in this case, 6). And the second keypress is echoed in the SQL editor:

    airflow=# SELECT * FROM dag_schedule_dataset_reference;
    airflow=# 6

• SIGINT isn't registered to cancel editing a query Example:

  • Write a query:

    airflow=# SELECT * FROM dag_messedupspelling

  • Then try to cancel/reset it with Ctrl+C:

    airflow=# SELECT * FROM dag_messedupspelling
    airflow-#

    (nothing happens) In normal psql this will reset the query editor (you can tell because the airflow-# prompt changes back to airflow=#).

Operating System

macOS 11.6.8 Big Sur

Versions of Apache Airflow Providers

N/A

Deployment

Other Docker-based deployment

Deployment details

Breeze

Anything else

This happens every time.

Apologies if this isn't the place to report Breeze issues.

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[uranusjr]

This is from one pane, right? I wonder if this is a Tmux issue (and if that’s the case there’s not much we can do). Can you try entering the shell (with breeze shell or even plain docker run) and run tmux manually, and see if this happens to you as well?

(This could also be related to terminal configuration; terminal emulation is pesky work.)

[potiuk]

This is neither breeze nor tmux problem. This is problem with airflow db shell for postgres and pass-through to pysql which should likely be improved.

You got the exact same behaviour when you use local virtualenv without either of the breeze/tmux (but you can reproduce it easily using breeze as the postgres database setup:

1. Create airflow virtualenv (an easy way is to activate empty venv and use ./scripts/tools/initialize_virtualenv.py - it will use constraints and you can choose which extras to install)
2. run breeze --backend postgres --db-reset (this will start and reset the postgres DB as well)
3. exit breeze
4. Setup your local venv airflow to connect to the Postgres db of Breeze:

   export AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://postgres:airflow@127.0.0.1:25433/airflow

   (you can see Postgres port number user/pasword used printed in the cheathsheet when you start breeze or find it in the docs) 5) Run airflow db shell

Observe the very same behaviour.

[potiuk]

I updated the description to clarify it. Fancy to attempt to fix it @blag ? I think it is the matter of how subprocess is used to run psql

[potiuk]

BTW. I jsut tested it additionally running this in start-airflow pane:

pysql -h postgres -U postgres

(password airflow) Which is equivalent of what breeze db shell does for postgres,

And it behaves as expected - so this is definitely airflow db shell that introduces it

[uranusjr]

Tangentally I’m wondering if it’s a good idea to use dbcli tools for airflow db shell.

[potiuk]

Looks cool. Definitely worth trying how well it works in airflow db. It will likely increase the dependency size, so possibly it should be something that is installed as "extra" ?

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.

[blag]

❌ Still broken - Scrollback editing is messed up - indentation and text wrapping are off/wrong ✅ Fixed...? - Some things require an additional keypress ❌ Still broken - SIGINT isn't registered to cancel editing a query

[potiuk]

Sure. Marked it as good first issue - maybe you would like to fix it @blag ? Otherwise it will have to wait for someone who would like to fix it.

[renzepost]

I can fix the SIGINT issue. I think the problem is in the fact that the TTY is set to raw mode, so in order to send the SIGINT signal to the subprocess, you need to listen for character code 3 in the input data.

The other issues mentioned here are quite puzzling for me though.

[potiuk]

If y ou can move it forward, that would be cool.

[potiuk]

@blag -> do you still the editing issue? Maybe you could take a stab at it and see if it can be fixed as well. Seems that with @renzepost we fixed signal propagation.

[blag]

❌ Still broken - Scrollback editing is messed up - indentation and text wrapping are off/wrong ❌ Still broken - Some things require an additional keypress, including Ctrl+D to exit a DB prompt (with both psql and sqlite) ✅ Fixed - SIGINT isn't registered to cancel editing a query

[blag]

Thank you @renzepost!

[renzepost]

Cool! If I have the time, I'll try to dig a little deeper in the other issues as well.