search

aws-samples / emr-serverless-samples

Example code for running Spark and Hive jobs on EMR Serverless.
https://aws.amazon.com/emr/serverless/
MIT No Attribution
155 stars 78 forks source link

Version mismatch in 'airflow/setup.py' #19

Closed fventuri-availity closed 2 years ago

fventuri-availity commented 2 years ago

I just noticed that the file 'setup.py' in the 'airflow' directory has still version 0.0.2 (https://github.com/aws-samples/emr-serverless-samples/blob/v0.0.4-preview/airflow/setup.py#L5), while the current version should be 0.0.4.

You may also want to look into updating the version numbers for the dependencies 'boto3' and 'botocore'.

Franco

dacort commented 2 years ago

Hi @fventuri-availity - Thanks for filing this, I happened to notice it recently as well.

Given that the EMR Serverless operator will be part of the official Amazon provider in Airflow (see https://github.com/apache/airflow/pull/25324 ), I will likely make one final release here and then point folks to that release when it's ready. :)

I'll look into bumping boto3/botocore, though - those versions were chosen as that is when EMR Serverless support was introduced, so wanted to ensure people were running at least those versions. Maybe I could do something along the lines of the following for boto3, for example:

boto3>=1.23.9, == 1.*

Although, hilariously I tried that locally and it downloaded every 1.24 version before settling on 1.23.10. 🤔

fventuri-availity commented 2 years ago

Thanks for your quick reply; since it looks like the EMR Serverless PR that you mentioned was merged into what looks like will become Airflow 2.4.0, while Amazon MWAA is still running Airflow version 2.2.2, I suspect it'll be a while before the operators in that specific PR will be available to us, so for now I think we'll be using your latest version.

Thanks for all your work, Franco

dacort commented 2 years ago

Fortunately, the providers are released on a regular cadence separately from the main Airflow release and just yesterday(!) v5.0.0 of the Amazon provider was released.

More details about how to use that can be found here: https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/index.html I've confirmed it works with OSS Airflow v2.2.2.

That said (😢), it looks like there's an issue trying to upgrade the default amazon provider package in MWAA based on some conversation I saw in the #airflow-aws slack channel.

Because of that, I'll pull in the latest changes, bump the version to 1.0 and make a release here in the next day or two. :)