Unable to retrieve tablefrom AWS Glue!

[iMilad]

the S3 bucket is empty and when I took a look at the Lambda, I saw this error, handled as warning! the main reason that I ended up to debug is error I get in Grafana

Any suggestion what could be the issue?

2024-05-02 12:19:00 3a0adf03-417f-4df3-991b-35cd2fae2363 WARN DynamoDBMetadataHandler:250 - doGetTable: Unable to retrieve table batch-op-dashboard-batchjobdata-41yi200clapo from AWSGlue in database/schema default. Falling back to schema inference. If inferred schema is incorrect, create a matching table in Glue to define schema (see README) com.amazonaws.services.glue.model.EntityNotFoundException: Entity Not Found (Service: AWSGlue; Status Code: 400; Error Code: EntityNotFoundException; Request ID: 03a04a20-5584-43a8-bf70-fcff1ce3cf52; Proxy: null)

I will test also the feature branch!

[mhuguesaws]

Did you run an AWS Batch job? It is required to run AWS Batch job to populate the DynamoDB that will be then use by Glue to build the table structure.

[iMilad]

ok, let me check and I will come back to you!

[iMilad]

yes we did, but just in case I ran a sample batch job again

the wizard** job queue is the new one

[mhuguesaws]

Can you check the content of the DynamoDB table BatchJobData-XXXX and report?

[iMilad]

contains the jobs, same for the other table batch-op-dashboard-ECSInstancexxxxx

[mhuguesaws]

Looks good. Let me think about next steps.

[iMilad]

I think, there is something wrong in configuration, because Lambda check for the other tables rather than these two, and the other table is empty! I have to re-check the deployment

[iMilad]

Does Lambda tries to read from all tables existed in DynamoDB?

[iMilad]

Ok, S3 now has the data, and Lambda redirect to the correct table BatchJobData-xxx , the issue was mentioning Default database in Grafana DataSource, but still Lambda has an error like before even with correct table

[mhuguesaws]

You can use more or less the same query from Grafana in amazon athena and see if you get a result. You will need to remove grafana specific timefilter and variable. That will remove Grafana as a component. Also sometime Grafana need a hard refresh in the browser to work correctly.

[iMilad]

OK, Nice! looks like I am getting closer. the filter was the key at least to see the jobs, but now I have to deal with the rest of dashboard as it does not show the metrics

[mhuguesaws]

You can see Amazon EC2 metrics for all jobs by selecting an instance id (click on the blue link) of a particular job. You can see a particular job metric with container insights by selecting the job.

[iMilad]

I have to take a look at the Dashboard code for the containers metrics as it does not work when you click on instance id, and see the same error as before in lambda. listing the job is working pretty fast but that's all so far.