search

awslabs / aws-glue-libs

AWS Glue Libraries are additions and enhancements to Spark for ETL operations.
Other
634 stars 298 forks source link

AWS Glue docker image com.amazonaws.auth.DefaultAWSCredentialsProviderChain does not implement AWSCredentialsProvider error on CREATE TABLE #130

Open JeevaTM opened 2 years ago

JeevaTM commented 2 years ago

I want to set local AWS Glue development environment. I created Glue Catalog Database.

I created an IAM user with full S3, Glue, Athena access. Granted the IAM user with super permission for Glue Catalog database in AWS Lake Formation.

I started a Glue notebook and ran the following script:

  import sys, os
    from awsglue.transforms import *
    from awsglue.utils import getResolvedOptions
    from pyspark.context import SparkContext
    from awsglue.context import GlueContext
    from awsglue.job import Job

    sc = SparkContext.getOrCreate()
    glueContext = GlueContext(sc)
    spark = glueContext.spark_session
    job = Job(glueContext)

    spark.sql("SHOW DATABASES").show()

    spark.sql("""
    CREATE TABLE IF NOT EXISTS db.jeeva_demo ( id int, name string )
    location  's3://bucket/warehouse/db/jeeva_demo';
    """)

    job.commit()

It successfully displayed the databases and created 'jeeva_demo' table in 'db' Glue Catalog database.

Now I wanted to setup same Glue environment locally to start working on it.

I pulled the docker image amazon/aws-glue-libs:glue_libs_3.0.0_image_01

Created a local folder glue. My test scripts reside in glue folder.

I ran the following command to start the Glue image

docker run -u 0 -it -v /home/jeeva/glue:/home/glue_user/workspace/ -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e AWS_PROFILE=$PROFILE_NAME -e AWS_REGION=$AWS_REGION -e AWS_EC2_METADATA_DISABLED=true -e DISABLE_SSL=true -p 4040:4040 -p 18080:18080 --name glue_pyspark amazon/aws-glue-libs:glue_libs_3.0.0_image_01

I am sitting behind organization firewall so I update Java keystore with zscaler.crt and did update-ca-trust with ca-bundle.pem.

I ran the same script I ran on AWS Glue locally

import sys, os
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

sc = SparkContext.getOrCreate()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)

# TO-DO
df = spark.sql("SHOW DATABASES")

spark.sql("""
CREATE TABLE IF NOT EXISTS db.jeeva_demo_local ( id int, name string )
location  's3://bucket/warehouse/db/jeeva_demo_local';
""")

job.commit()

The variable df on debug showed the databases I gave IAM user permission in AWS Lake Formation.

Script would run until it is on CREATE TABLE line and throws the error.

    22/04/08 12:54:44 ERROR log: Converting exception to MetaException
    Traceback (most recent call last):
      File "/home/glue_user/workspace/check.py", line 19, in <module>
        """)
      File "/home/glue_user/spark/python/pyspark/sql/session.py", line 723, in sql
        return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
      File "/home/glue_user/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__
      File "/home/glue_user/spark/python/pyspark/sql/utils.py", line 117, in deco
        raise converted from None
    pyspark.sql.utils.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.io.IOException Class class com.amazonaws.auth.DefaultAWSCredentialsProviderChain does not implement AWSCredentialsProvider)

I assumed this was error due to AWS Java SDK - S3 missing so I added spark.jars.packages com.amazonaws:aws-java-sdk:1.11.1000 in /home/glue_user/spark/conf/spark-defaults.conf and ran the python script again.

Spark did download the dependencies but I still have the same error. What am I missing here?

Full trace:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/glue_user/spark/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/glue_user/aws-glue-libs/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
:: loading settings :: url = jar:file:/home/glue_user/spark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /root/.ivy2/cache
The jars for the packages stored in: /root/.ivy2/jars
com.amazonaws#aws-java-sdk added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-a246b699-881a-4547-8638-c2305601701c;1.0
        confs: [default]
        found com.amazonaws#aws-java-sdk;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-lookoutequipment;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-core;1.11.1000 in central
        found commons-logging#commons-logging;1.1.3 in central
        found commons-codec#commons-codec;1.15 in central
        found org.apache.httpcomponents#httpclient;4.5.13 in central
        found org.apache.httpcomponents#httpcore;4.4.13 in central
        found software.amazon.ion#ion-java;1.0.2 in central
        found com.fasterxml.jackson.core#jackson-databind;2.6.7.4 in central
        found com.fasterxml.jackson.core#jackson-annotations;2.6.0 in central
        found com.fasterxml.jackson.core#jackson-core;2.6.7 in central
        found com.fasterxml.jackson.dataformat#jackson-dataformat-cbor;2.6.7 in central
        found joda-time#joda-time;2.8.1 in central
        found com.amazonaws#jmespath-java;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-mgn;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-lookoutmetrics;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-fis;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-lexmodelsv2;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-lexruntimev2;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-wellarchitected;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-location;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-iotdeviceadvisor;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-iotfleethub;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-iotwireless;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-greengrassv2;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-prometheus;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-auditmanager;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-healthlake;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-sagemakeredgemanager;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-emrcontainers;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-customerprofiles;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-lookoutforvision;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-connectcontactlens;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-amplifybackend;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-ecrpublic;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-appintegrations;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-sagemakerfeaturestoreruntime;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-devopsguru;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-mwaa;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-networkfirewall;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-appregistry;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-gluedatabrew;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-s3outposts;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-timestreamquery;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-timestreamwrite;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-ssoadmin;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-redshiftdataapi;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-appflow;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-identitystore;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-braket;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-ivs;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-honeycode;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-codeartifact;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-macie2;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-iotsitewise;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-synthetics;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-codestarconnections;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-detective;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-kinesisvideosignalingchannels;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-ebs;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-augmentedairuntime;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-outposts;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-codeguruprofiler;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-codegurureviewer;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-frauddetector;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-kendra;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-networkmanager;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-computeoptimizer;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-accessanalyzer;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-schemas;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-imagebuilder;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-elasticinference;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-iotsecuretunneling;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-appconfig;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-wafv2;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-connectparticipant;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-migrationhubconfig;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-dataexchange;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-sesv2;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-marketplacecatalog;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-ssooidc;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-sso;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-savingsplans;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-codestarnotifications;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-workmailmessageflow;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-qldbsession;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-qldb;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-forecastquery;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-forecast;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-lakeformation;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-eventbridge;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-ec2instanceconnect;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-applicationinsights;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-servicequotas;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-personalizeevents;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-personalize;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-personalizeruntime;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-ioteventsdata;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-iotevents;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-iotthingsgraph;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-groundstation;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-mediapackagevod;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-managedblockchain;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-textract;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-worklink;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-backup;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-docdb;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-apigatewayv2;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-apigatewaymanagementapi;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-kafka;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-appmesh;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-licensemanager;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-securityhub;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-fsx;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-mediaconnect;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-kinesisanalyticsv2;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-comprehendmedical;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-globalaccelerator;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-transfer;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-datasync;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-robomaker;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-amplify;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-quicksight;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-rdsdata;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-route53resolver;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-ram;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-s3control;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-s3;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-kms;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-pinpointsmsvoice;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-pinpointemail;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-chime;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-signer;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-dlm;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-macie;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-eks;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-mediatailor;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-neptune;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-pi;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-iot1clickprojects;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-iot1clickdevices;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-iotanalytics;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-acmpca;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-secretsmanager;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-fms;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-connect;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-transcribe;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-autoscalingplans;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-workmail;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-servicediscovery;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-cloud9;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-serverlessapplicationrepository;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-alexaforbusiness;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-resourcegroups;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-comprehend;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-translate;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-sagemaker;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-iotjobsdataplane;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-sagemakerruntime;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-kinesisvideo;1.11.1000 in central
        found io.netty#netty-codec-http;4.1.61.Final in central
        found io.netty#netty-common;4.1.61.Final in central
        found io.netty#netty-buffer;4.1.61.Final in central
        found io.netty#netty-transport;4.1.61.Final in central
        found io.netty#netty-resolver;4.1.61.Final in central
        found io.netty#netty-codec;4.1.61.Final in central
        found io.netty#netty-handler;4.1.61.Final in central
        found com.amazonaws#aws-java-sdk-appsync;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-guardduty;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-mq;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-mediaconvert;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-mediastore;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-mediastoredata;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-medialive;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-mediapackage;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-costexplorer;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-pricing;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-mobile;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-cloudhsmv2;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-glue;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-migrationhub;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-dax;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-greengrass;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-athena;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-marketplaceentitlement;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-codestar;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-lexmodelbuilding;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-resourcegroupstaggingapi;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-pinpoint;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-xray;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-opsworkscm;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-support;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-simpledb;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-servicecatalog;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-servermigration;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-simpleworkflow;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-storagegateway;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-route53;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-importexport;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-sts;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-sqs;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-rds;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-redshift;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-elasticbeanstalk;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-glacier;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-sns;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-iam;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-datapipeline;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-elasticloadbalancing;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-elasticloadbalancingv2;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-emr;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-elasticache;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-elastictranscoder;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-ec2;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-dynamodb;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-budgets;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-cloudtrail;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-cloudwatch;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-logs;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-events;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-cognitoidentity;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-cognitosync;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-directconnect;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-cloudformation;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-cloudfront;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-clouddirectory;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-kinesis;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-opsworks;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-ses;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-autoscaling;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-cloudsearch;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-cloudwatchmetrics;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-codedeploy;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-codepipeline;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-config;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-lambda;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-ecs;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-ecr;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-cloudhsm;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-ssm;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-workspaces;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-machinelearning;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-directory;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-efs;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-codecommit;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-devicefarm;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-elasticsearch;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-waf;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-marketplacecommerceanalytics;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-inspector;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-iot;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-api-gateway;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-acm;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-gamelift;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-dms;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-marketplacemeteringservice;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-cognitoidp;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-discovery;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-applicationautoscaling;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-snowball;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-rekognition;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-polly;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-lightsail;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-stepfunctions;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-health;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-costandusagereport;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-codebuild;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-appstream;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-shield;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-batch;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-lex;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-mechanicalturkrequester;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-organizations;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-workdocs;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-models;1.11.1000 in central
        found com.amazonaws#aws-java-sdk-swf-libraries;1.11.22 in central
:: resolution report :: resolve 36048ms :: artifacts dl 262ms
        :: modules in use:
        com.amazonaws#aws-java-sdk;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-accessanalyzer;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-acm;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-acmpca;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-alexaforbusiness;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-amplify;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-amplifybackend;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-api-gateway;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-apigatewaymanagementapi;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-apigatewayv2;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-appconfig;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-appflow;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-appintegrations;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-applicationautoscaling;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-applicationinsights;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-appmesh;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-appregistry;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-appstream;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-appsync;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-athena;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-auditmanager;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-augmentedairuntime;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-autoscaling;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-autoscalingplans;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-backup;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-batch;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-braket;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-budgets;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-chime;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-cloud9;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-clouddirectory;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-cloudformation;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-cloudfront;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-cloudhsm;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-cloudhsmv2;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-cloudsearch;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-cloudtrail;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-cloudwatch;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-cloudwatchmetrics;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-codeartifact;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-codebuild;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-codecommit;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-codedeploy;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-codeguruprofiler;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-codegurureviewer;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-codepipeline;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-codestar;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-codestarconnections;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-codestarnotifications;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-cognitoidentity;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-cognitoidp;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-cognitosync;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-comprehend;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-comprehendmedical;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-computeoptimizer;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-config;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-connect;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-connectcontactlens;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-connectparticipant;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-core;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-costandusagereport;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-costexplorer;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-customerprofiles;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-dataexchange;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-datapipeline;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-datasync;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-dax;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-detective;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-devicefarm;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-devopsguru;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-directconnect;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-directory;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-discovery;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-dlm;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-dms;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-docdb;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-dynamodb;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-ebs;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-ec2;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-ec2instanceconnect;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-ecr;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-ecrpublic;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-ecs;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-efs;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-eks;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-elasticache;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-elasticbeanstalk;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-elasticinference;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-elasticloadbalancing;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-elasticloadbalancingv2;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-elasticsearch;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-elastictranscoder;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-emr;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-emrcontainers;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-eventbridge;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-events;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-fis;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-fms;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-forecast;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-forecastquery;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-frauddetector;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-fsx;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-gamelift;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-glacier;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-globalaccelerator;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-glue;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-gluedatabrew;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-greengrass;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-greengrassv2;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-groundstation;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-guardduty;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-health;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-healthlake;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-honeycode;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-iam;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-identitystore;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-imagebuilder;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-importexport;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-inspector;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-iot;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-iot1clickdevices;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-iot1clickprojects;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-iotanalytics;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-iotdeviceadvisor;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-iotevents;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-ioteventsdata;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-iotfleethub;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-iotjobsdataplane;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-iotsecuretunneling;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-iotsitewise;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-iotthingsgraph;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-iotwireless;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-ivs;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-kafka;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-kendra;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-kinesis;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-kinesisanalyticsv2;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-kinesisvideo;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-kinesisvideosignalingchannels;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-kms;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-lakeformation;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-lambda;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-lex;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-lexmodelbuilding;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-lexmodelsv2;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-lexruntimev2;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-licensemanager;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-lightsail;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-location;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-logs;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-lookoutequipment;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-lookoutforvision;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-lookoutmetrics;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-machinelearning;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-macie;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-macie2;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-managedblockchain;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-marketplacecatalog;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-marketplacecommerceanalytics;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-marketplaceentitlement;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-marketplacemeteringservice;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-mechanicalturkrequester;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-mediaconnect;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-mediaconvert;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-medialive;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-mediapackage;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-mediapackagevod;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-mediastore;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-mediastoredata;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-mediatailor;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-mgn;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-migrationhub;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-migrationhubconfig;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-mobile;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-models;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-mq;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-mwaa;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-neptune;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-networkfirewall;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-networkmanager;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-opsworks;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-opsworkscm;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-organizations;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-outposts;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-personalize;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-personalizeevents;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-personalizeruntime;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-pi;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-pinpoint;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-pinpointemail;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-pinpointsmsvoice;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-polly;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-pricing;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-prometheus;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-qldb;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-qldbsession;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-quicksight;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-ram;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-rds;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-rdsdata;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-redshift;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-redshiftdataapi;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-rekognition;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-resourcegroups;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-resourcegroupstaggingapi;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-robomaker;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-route53;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-route53resolver;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-s3;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-s3control;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-s3outposts;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-sagemaker;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-sagemakeredgemanager;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-sagemakerfeaturestoreruntime;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-sagemakerruntime;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-savingsplans;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-schemas;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-secretsmanager;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-securityhub;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-serverlessapplicationrepository;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-servermigration;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-servicecatalog;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-servicediscovery;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-servicequotas;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-ses;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-sesv2;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-shield;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-signer;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-simpledb;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-simpleworkflow;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-snowball;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-sns;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-sqs;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-ssm;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-sso;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-ssoadmin;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-ssooidc;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-stepfunctions;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-storagegateway;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-sts;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-support;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-swf-libraries;1.11.22 from central in [default]
        com.amazonaws#aws-java-sdk-synthetics;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-textract;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-timestreamquery;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-timestreamwrite;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-transcribe;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-transfer;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-translate;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-waf;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-wafv2;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-wellarchitected;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-workdocs;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-worklink;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-workmail;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-workmailmessageflow;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-workspaces;1.11.1000 from central in [default]
        com.amazonaws#aws-java-sdk-xray;1.11.1000 from central in [default]
        com.amazonaws#jmespath-java;1.11.1000 from central in [default]
        com.fasterxml.jackson.core#jackson-annotations;2.6.0 from central in [default]
        com.fasterxml.jackson.core#jackson-core;2.6.7 from central in [default]
        com.fasterxml.jackson.core#jackson-databind;2.6.7.4 from central in [default]
        com.fasterxml.jackson.dataformat#jackson-dataformat-cbor;2.6.7 from central in [default]
        commons-codec#commons-codec;1.15 from central in [default]
        commons-logging#commons-logging;1.1.3 from central in [default]
        io.netty#netty-buffer;4.1.61.Final from central in [default]
        io.netty#netty-codec;4.1.61.Final from central in [default]
        io.netty#netty-codec-http;4.1.61.Final from central in [default]
        io.netty#netty-common;4.1.61.Final from central in [default]
        io.netty#netty-handler;4.1.61.Final from central in [default]
        io.netty#netty-resolver;4.1.61.Final from central in [default]
        io.netty#netty-transport;4.1.61.Final from central in [default]
        joda-time#joda-time;2.8.1 from central in [default]
        org.apache.httpcomponents#httpclient;4.5.13 from central in [default]
        org.apache.httpcomponents#httpcore;4.4.13 from central in [default]
        software.amazon.ion#ion-java;1.0.2 from central in [default]
        :: evicted modules:
        commons-logging#commons-logging;1.2 by [commons-logging#commons-logging;1.1.3] in [default]
        commons-codec#commons-codec;1.11 by [commons-codec#commons-codec;1.15] in [default]
        com.amazonaws#aws-java-sdk-simpleworkflow;1.11.22 by [com.amazonaws#aws-java-sdk-simpleworkflow;1.11.1000] in [default]
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |  279  |   0   |   0   |   3   ||  276  |   0   |
        ---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-a246b699-881a-4547-8638-c2305601701c
        confs: [default]
        0 artifacts copied, 276 already retrieved (0kB/277ms)
22/04/08 12:53:17 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/04/08 12:54:02 ERROR Inbox: Ignoring error
java.lang.NullPointerException
        at org.apache.spark.storage.BlockManagerMasterEndpoint.org$apache$spark$storage$BlockManagerMasterEndpoint$$register(BlockManagerMasterEndpoint.scala:533)
        at org.apache.spark.storage.BlockManagerMasterEndpoint$$anonfun$receiveAndReply$1.applyOrElse(BlockManagerMasterEndpoint.scala:116)
        at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:103)
        at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)
        at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
        at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
        at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
22/04/08 12:54:02 WARN Executor: Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Exception thrown in awaitResult: 
        at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)
        at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
        at org.apache.spark.rpc.RpcEndpointRef.askSync(RpcEndpointRef.scala:103)
        at org.apache.spark.rpc.RpcEndpointRef.askSync(RpcEndpointRef.scala:87)
        at org.apache.spark.storage.BlockManagerMaster.registerBlockManager(BlockManagerMaster.scala:87)
        at org.apache.spark.storage.BlockManager.reregister(BlockManager.scala:589)
        at org.apache.spark.executor.Executor.reportHeartBeat(Executor.scala:1000)
        at org.apache.spark.executor.Executor.$anonfun$heartbeater$1(Executor.scala:212)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1996)
        at org.apache.spark.Heartbeater$$anon$1.run(Heartbeater.scala:46)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.NullPointerException
        at org.apache.spark.storage.BlockManagerMasterEndpoint.org$apache$spark$storage$BlockManagerMasterEndpoint$$register(BlockManagerMasterEndpoint.scala:533)
        at org.apache.spark.storage.BlockManagerMasterEndpoint$$anonfun$receiveAndReply$1.applyOrElse(BlockManagerMasterEndpoint.scala:116)
        at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:103)
        at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)
        at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
        at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
        at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        ... 3 more
22/04/08 12:54:31 WARN EC2MetadataUtils: Unable to retrieve the requested metadata (/latest/dynamic/instance-identity/document). EC2 Instance Metadata Service is disabled
com.amazonaws.AmazonClientException: EC2 Instance Metadata Service is disabled
        at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:65)
        at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)
        at com.amazonaws.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:403)
        at com.amazonaws.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:372)
        at com.amazonaws.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:368)
        at com.amazonaws.util.EC2MetadataUtils.getEC2InstanceRegion(EC2MetadataUtils.java:283)
        at com.amazonaws.regions.Regions.getCurrentRegion(Regions.java:109)
        at com.amazonaws.glue.catalog.metastore.AWSGlueClientFactory.newClient(AWSGlueClientFactory.java:64)
        at com.amazonaws.glue.catalog.metastore.AWSCatalogMetastoreClient.<init>(AWSCatalogMetastoreClient.java:142)
        at com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory.createMetaStoreClient(AWSGlueDataCatalogHiveClientFactory.java:20)
        at org.apache.hadoop.hive.ql.metadata.HiveUtils.createMetaStoreClient(HiveUtils.java:507)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3746)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3726)
        at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3988)
        at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:251)
        at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:234)
        at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:402)
        at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:335)
        at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:315)
        at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:291)
        at org.apache.spark.sql.hive.client.HiveClientImpl.client(HiveClientImpl.scala:257)
        at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:283)
        at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:224)
        at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:223)
        at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:273)
        at org.apache.spark.sql.hive.client.HiveClientImpl.databaseExists(HiveClientImpl.scala:384)
        at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$databaseExists$1(HiveExternalCatalog.scala:249)
        at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
        at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:105)
        at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:249)
        at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:135)
        at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:125)
        at org.apache.spark.sql.hive.HiveSessionStateBuilder.externalCatalog(HiveSessionStateBuilder.scala:44)
        at org.apache.spark.sql.hive.HiveSessionStateBuilder.$anonfun$catalog$1(HiveSessionStateBuilder.scala:51)
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.externalCatalog$lzycompute(SessionCatalog.scala:98)
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.externalCatalog(SessionCatalog.scala:98)
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.listDatabases(SessionCatalog.scala:270)
        at org.apache.spark.sql.execution.datasources.v2.V2SessionCatalog.listNamespaces(V2SessionCatalog.scala:203)
        at org.apache.spark.sql.execution.datasources.v2.ShowNamespacesExec.run(ShowNamespacesExec.scala:43)
        at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:40)
        at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:40)
        at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:46)
        at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:229)
        at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3724)
        at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
        at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
        at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:110)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:135)
        at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
        at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:135)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:253)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:134)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)
        at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3722)
        at org.apache.spark.sql.Dataset.<init>(Dataset.scala:229)
        at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
        at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
        at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:615)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:610)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:750)
22/04/08 12:54:38 WARN ResolveSessionCatalog: A Hive serde table will be created as there is no table provider specified. You can set spark.sql.legacy.createHiveTableByDefault to false so that native data source table will be created instead.
22/04/08 12:54:40 WARN SessionState: METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.
22/04/08 12:54:40 WARN EC2MetadataUtils: Unable to retrieve the requested metadata (/latest/dynamic/instance-identity/document). EC2 Instance Metadata Service is disabled
com.amazonaws.AmazonClientException: EC2 Instance Metadata Service is disabled
        at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:65)
        at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)
        at com.amazonaws.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:403)
        at com.amazonaws.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:372)
        at com.amazonaws.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:368)
        at com.amazonaws.util.EC2MetadataUtils.getEC2InstanceRegion(EC2MetadataUtils.java:283)
        at com.amazonaws.regions.Regions.getCurrentRegion(Regions.java:109)
        at com.amazonaws.glue.catalog.metastore.AWSGlueClientFactory.newClient(AWSGlueClientFactory.java:64)
        at com.amazonaws.glue.catalog.metastore.AWSCatalogMetastoreClient.<init>(AWSCatalogMetastoreClient.java:142)
        at com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory.createMetaStoreClient(AWSGlueDataCatalogHiveClientFactory.java:20)
        at org.apache.hadoop.hive.ql.metadata.HiveUtils.createMetaStoreClient(HiveUtils.java:507)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3746)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3726)
        at org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:917)
        at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:881)
        at org.apache.hadoop.hive.ql.session.SessionState.getAuthenticator(SessionState.java:1483)
        at org.apache.hadoop.hive.ql.session.SessionState.getUserFromAuthenticator(SessionState.java:1154)
        at org.apache.hadoop.hive.ql.metadata.Table.getEmptyTable(Table.java:180)
        at org.apache.hadoop.hive.ql.metadata.Table.<init>(Table.java:122)
        at org.apache.spark.sql.hive.client.HiveClientImpl$.toHiveTable(HiveClientImpl.scala:1078)
        at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$createTable$1(HiveClientImpl.scala:543)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:291)
        at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:224)
        at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:223)
        at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:273)
        at org.apache.spark.sql.hive.client.HiveClientImpl.createTable(HiveClientImpl.scala:541)
        at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$createTable$1(HiveExternalCatalog.scala:312)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:105)
        at org.apache.spark.sql.hive.HiveExternalCatalog.createTable(HiveExternalCatalog.scala:270)
        at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.createTable(ExternalCatalogWithListener.scala:94)
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.createTable(SessionCatalog.scala:346)
        at org.apache.spark.sql.execution.command.CreateTableCommand.run(tables.scala:166)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
        at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:229)
        at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3724)
        at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
        at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
        at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:110)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:135)
        at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
        at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:135)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:253)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:134)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)
        at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3722)
        at org.apache.spark.sql.Dataset.<init>(Dataset.scala:229)
        at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
        at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
        at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:615)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:610)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:750)
22/04/08 12:54:42 WARN EC2MetadataUtils: Unable to retrieve the requested metadata (/latest/dynamic/instance-identity/document). EC2 Instance Metadata Service is disabled
com.amazonaws.AmazonClientException: EC2 Instance Metadata Service is disabled
        at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:65)
        at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)
        at com.amazonaws.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:403)
        at com.amazonaws.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:372)
        at com.amazonaws.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:368)
        at com.amazonaws.util.EC2MetadataUtils.getEC2InstanceRegion(EC2MetadataUtils.java:283)
        at com.amazonaws.regions.Regions.getCurrentRegion(Regions.java:109)
        at com.amazonaws.glue.catalog.metastore.AWSGlueClientFactory.newClient(AWSGlueClientFactory.java:64)
        at com.amazonaws.glue.catalog.metastore.AWSCatalogMetastoreClient.<init>(AWSCatalogMetastoreClient.java:142)
        at com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory.createMetaStoreClient(AWSGlueDataCatalogHiveClientFactory.java:20)
        at org.apache.hadoop.hive.ql.metadata.HiveUtils.createMetaStoreClient(HiveUtils.java:507)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3746)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3726)
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:866)
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:881)
        at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$createTable$1(HiveClientImpl.scala:543)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:291)
        at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:224)
        at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:223)
        at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:273)
        at org.apache.spark.sql.hive.client.HiveClientImpl.createTable(HiveClientImpl.scala:541)
        at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$createTable$1(HiveExternalCatalog.scala:312)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:105)
        at org.apache.spark.sql.hive.HiveExternalCatalog.createTable(HiveExternalCatalog.scala:270)
        at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.createTable(ExternalCatalogWithListener.scala:94)
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.createTable(SessionCatalog.scala:346)
        at org.apache.spark.sql.execution.command.CreateTableCommand.run(tables.scala:166)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
        at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:229)
        at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3724)
        at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
        at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
        at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:110)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:135)
        at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
        at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:135)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:253)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:134)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)
        at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3722)
        at org.apache.spark.sql.Dataset.<init>(Dataset.scala:229)
        at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
        at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
        at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:615)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:610)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:750)
22/04/08 12:54:44 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
22/04/08 12:54:44 ERROR log: Got exception: java.io.IOException Class class com.amazonaws.auth.DefaultAWSCredentialsProviderChain does not implement AWSCredentialsProvider
java.io.IOException: Class class com.amazonaws.auth.DefaultAWSCredentialsProviderChain does not implement AWSCredentialsProvider
        at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:662)
        at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:605)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:260)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3358)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3407)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3375)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:486)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
        at org.apache.hadoop.hive.metastore.Warehouse.getFs(Warehouse.java:105)
        at org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:131)
        at org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:137)
        at com.amazonaws.glue.catalog.metastore.GlueMetastoreClientDelegate.validateNewTableAndCreateDirectory(GlueMetastoreClientDelegate.java:685)
        at com.amazonaws.glue.catalog.metastore.GlueMetastoreClientDelegate.createTable(GlueMetastoreClientDelegate.java:388)
        at com.amazonaws.glue.catalog.metastore.AWSCatalogMetastoreClient.createTable(AWSCatalogMetastoreClient.java:608)
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:866)
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:881)
        at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$createTable$1(HiveClientImpl.scala:543)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:291)
        at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:224)
        at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:223)
        at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:273)
        at org.apache.spark.sql.hive.client.HiveClientImpl.createTable(HiveClientImpl.scala:541)
        at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$createTable$1(HiveExternalCatalog.scala:312)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:105)
        at org.apache.spark.sql.hive.HiveExternalCatalog.createTable(HiveExternalCatalog.scala:270)
        at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.createTable(ExternalCatalogWithListener.scala:94)
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.createTable(SessionCatalog.scala:346)
        at org.apache.spark.sql.execution.command.CreateTableCommand.run(tables.scala:166)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
        at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:229)
        at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3724)
        at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
        at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
        at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:110)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:135)
        at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
        at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:135)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:253)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:134)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)
        at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3722)
        at org.apache.spark.sql.Dataset.<init>(Dataset.scala:229)
        at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
        at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
        at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:615)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:610)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:750)
22/04/08 12:54:44 ERROR log: Converting exception to MetaException
Traceback (most recent call last):
  File "/home/glue_user/workspace/check.py", line 19, in <module>
    """)
  File "/home/glue_user/spark/python/pyspark/sql/session.py", line 723, in sql
    return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
  File "/home/glue_user/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__
  File "/home/glue_user/spark/python/pyspark/sql/utils.py", line 117, in deco
    raise converted from None
pyspark.sql.utils.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.io.IOException Class class com.amazonaws.auth.DefaultAWSCredentialsProviderChain does not implement AWSCredentialsProvider)
xmubeta commented 2 years ago

I hit the same issue. It seems to be related to Class loader for Hive to load credential class. Anyway, I cannot fix it without code, but I do find a workaround:

Create core-site.xml in aws-glue-libs/conf/, replace your AKSK in it.

cat aws-glue-libs/conf/core-site.xml

<configuration>
     <property>
         <name>fs.s3.impl</name>
         <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
         <description>s3a filesystem implementation</description>
     </property>

<property>
  <name>fs.s3a.access.key</name>
  <value>AK</value>
</property>
<property>
  <name>fs.s3a.secret.key</name>
  <value>SK</value>
</property>
</configuration>

Hopefully it works for you.

labbedaine commented 2 years ago

@JeevaTM I am facing the same problem. Did you find a resolution? I will investigate @xmubeta solution but not sure yet how to do that.

JeevaTM commented 2 years ago

@labbedaine I used to following Dockerfile content to work with Iceberg tables in Glue. It works pretty well

FROM amazon/aws-glue-libs:glue_libs_3.0.0_image_01 ENV AWS_ACCESS_KEY_ID="redacted" ENV AWS_SECRET_ACCESS_KEY="redacted" ENV DISABLE_SSL=true ENV AWS_CA_BUNDLE="/etc/pki/ca-trust/source/anchors/ca-bundle.pem"

USER 0

COPY . /home/glue_user/workspace/

COPY cred/ca-bundle.pem /etc/pki/ca-trust/source/anchors/ RUN update-ca-trust

COPY cred/zscaler.crt /home/glue_user/ WORKDIR /usr/lib/jvm/java-1.8.0-amazon-corretto.x86_64/jre/lib/security/ RUN keytool -import -keystore ./cacerts -trustcacerts -file /home/glue_user/zscaler.crt -storepass changeit -noprompt

WORKDIR /home/glue_user/spark/jars/ RUN wget https://repo1.maven.org/maven2/software/amazon/awssdk/bundle/2.15.40/bundle-2.15.40.jar RUN wget https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark3-runtime/0.12.0/iceberg-spark3-runtime-0.12.0.jar RUN wget https://repo1.maven.org/maven2/software/amazon/awssdk/url-connection-client/2.15.40/url-connection-client-2.15.40.jar

WORKDIR /home/glue_user/aws-glue-libs/awsglue/scripts/ RUN cat /home/glue_user/workspace/cred/ca-bundle.pem >> /home/glue_user/.local/lib/python3.7/site-packages/certifi/cacert.pem RUN cat /home/glue_user/workspace/cred/ca-bundle.pem >> /usr/local/lib/python3.7/site-packages/certifi/cacert.pem

RUN chown -R glue_user /home/glue_user/

WORKDIR /home/glue_user/workspace/

USER 10000

cabundle.pem is Certificate authority and zscaler is my corporate proxy certificate. Update if required.

The jar file version on wget is necessary to work with Icebergs. Unfortunately, you cannot work with Iceberg 0.13.x with the Glue image, it comes with Spark 3.2 but 3.3 is required so if you write to Iceberg from this Image columns will be missing in Glue data catalog, and Athena preview. Quicksight cannot read the Iceberg even with 'query using athena option'.

All of the above goes for AWS Glue as well, not just the image.

I would divert you to use EMR instead, which comes with 0.13.x iceberg support and such.