astronomer / starship

Astronomer Starship can send your Airflow workloads to new places!
https://astronomer.github.io/starship
Other
27 stars 5 forks source link

Starship is not backwards-compatible with Airflow 2.2 #47

Closed fritz-astronomer closed 7 months ago

fritz-astronomer commented 1 year ago
[0m[91mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
flask-jwt-extended 3.25.1 requires PyJWT<2.0,>=1.6.4, but you have pyjwt 2.7.0 which is incompatible.
flask-appbuilder 3.3.3 requires PyJWT<2.0.0,>=1.7.1, but you have pyjwt 2.7.0 which is incompatible.
apache-airflow 1!2.1.4+astro.2 requires pyjwt<2, but you have pyjwt 2.7.0 which is incompatible.
fritz-astronomer commented 1 year ago

The latest version of astronomer-starship is only supported in Python 3.8 and above. However, they are using AC 2.1.4-2 which is built on top of Python 3.7 which is causing a conflict. To resolve the conflict I have asked them to try with astronomer-starship==0.3.5 which resolved the issue.

fritz-astronomer commented 1 year ago

This potentially works

pip install astronomer-starship cachetools==4.2.4 
pyjwt==1.7.1
fritz-astronomer commented 1 year ago

Specific fix documented here https://github.com/astronomer/starship#faq

fritz-astronomer commented 1 year ago

Potentially fixed in v1.1.0 via #40 , #41 , #42, and #43

fritz-astronomer commented 1 year ago

Most recent testing: RT8.8 ✅ RT5.4 ✅ RT4.2 ❌ can’t pause DAGs

fritz-astronomer commented 1 year ago

Possibly new issue - 4.2.9 Localhost -> 4.2.9 on Astro - can’t pause DAGs in target Airflow


 Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app
   response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1952, in full_dispatch_request
   rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1821, in handle_user_exception
   reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
   raise value
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request
   rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request
   return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.9/site-packages/airflow/www/auth.py", line 51, in decorated
   return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/astronomer/starship/main.py", line 343, in dag_cutover_row
   remote_airflow_client.set_dag_is_paused(
  File "/usr/local/lib/python3.9/site-packages/astronomer/starship/services/remote_airflow_client.py", line 143, in set_dag_is_paused
   r.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 960, in raise_for_status
   raise HTTPError(http_error_msg, response=self)
 requests.exceptions.HTTPError: 405 Client Error: Method Not Allowed for url: https://clkvh3b46003m01kbalgwwdcy.astronomer.run/dicash2x/api/v1/dags?dag_id_pattern=tutorial
 172.18.0.1 - - [10/Aug/2023:19:53:29 +0000] "POST /astromigration/component/row/dags/cll49pbo84339891u1jgicash2x/astro/tutorial/pause HTTP/1.1" 500 1551 "http://localhost:8080/astromigration/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36"```
fritz-astronomer commented 7 months ago

Closing as WONTFIX - Starship v2 has a full rewrite, which should fix this