Closed simonvdk closed 1 year ago
Lake Formation permissions are not supported in aws-glue-lib's Docker container. To use Lake Formation permission integrations, we recommend you to run the job on Glue job system (Glue jobs, or Glue Interactive Sessions).
I have an issue when using the
GlueContext.create_dynamic_frame_from_catalog
method in order to load a Glue Data Catalog table when Lakeformation is activated on that Glue Data Catalog.Steps to reproduce: (reproduced with both images
amazon/aws-glue-libs:glue_libs_2.0.0_image_01
andamazon/aws-glue-libs:glue_libs_3.0.0_image_01
)SELECT
andDESCRIBE
Lake Formation permissions on the table I want to loadResult:
The
create_dynamic_frame_from_catalog
call returns an 403 S3 access denied (see below for more details on traceback)Conclusions:
Insufficient Lake Formation permissions
)Note: From within the container, I was able to successfully query the same Glue table using Athena. This indicates that the error above is not due to how I launched my container or other configurations, as the Lake Formation credentials vending worked well with the Athena query. The issue hence comes from how the glue libs handles Lakeformation credentials vending when retrieving the table
Full traceback: