aelzeiny
commented
1 year ago Very kind words in regards to blowing everything out of the water. Thank you for your patience and understanding. Boto3 Authentication just defaults to whatever credentials are set on the machine per the AWS documentation. This is also how Airflow's core AWS libraries work, so hopefully no new patterns have been introduced.
Something you can do to debug this is hardcode credentials/aws profile (just temporarily). You can create a class, overwrite the start function to declare boto3's client the way you want it, and point to that new class in your Airflow CFG.
Not sure if it's an issue with the executor or boto or airflow itself. I'm setting up the scheduler and webserver in fargate, and when starting the container as root to do some file movement, and then switching to airflow user to run the scheduler; at startup when the batch executor gets initialised it's trying to contact the wrong instance metadata service (ec2) rather then fargate one. In contrast, when stating the container as airflow user, everything works fine and we can schedule the batch jobs.
Is there a way to force boto3 authentication in the executor config?
P.S. awesome job this executor, I tried literally everything else and it blows it out the water.