search

NASA-IMPACT / veda-pforge-job-runner

Apache Beam + EMR Serverless Job Runner for Pangeo Forge Recipes
2 stars 2 forks source link

Failing: NClimGrid #18

Closed ranchodeluxe closed 3 months ago

ranchodeluxe commented 7 months ago 
pangeo-forge-runner bake \
    --repo=https://github.com/ranchodeluxe/staged-recipes \
    --ref="nclimgrid" \
    --Bake.feedstock_subdir="recipes/noaa-nclimgrid" \
    -f config.py 
pangeo-forge-runner bake \
    --repo=https://github.com/ranchodeluxe/staged-recipes \
    --ref="nclimgrid" \
    --Bake.feedstock_subdir="recipes/noaa-nclimgrid" \
    -f config.py 
curl -X POST \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
-H "Authorization: token blablah" \
https://api.github.com/repos/NASA-IMPACT/veda-pforge-job-runner/actions/workflows/job-runner.yaml/dispatches \
-d '{"ref":"main", "inputs":{"repo":"https://github.com/ranchodeluxe/staged-recipes","ref":"nclimgrid","prune":"1","feedstock_subdir": "recipes/noaa-nclimgrid"}}'
curl -X POST \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
-H "Authorization: token blablah" \
https://api.github.com/repos/NASA-IMPACT/veda-pforge-job-runner/actions/workflows/job-runner.yaml/dispatches \
-d '{"ref":"main", "inputs":{"repo":"https://github.com/ranchodeluxe/staged-recipes","ref":"nclimgrid","prune":"1","feedstock_subdir": "recipes/noaa-nclimgrid"}}'
ranchodeluxe commented 7 months ago

failing without good errors: https://github.com/NASA-IMPACT/veda-pforge-job-runner/actions/runs/7441679947

ranchodeluxe commented 7 months ago

Better errors now that we fixed the memory stuff. We have a permission error:

 Found credentials from IAM Role: pangeo-forge-v3-nodegroup-role                                                                         
2024-01-08 23:47:08,743 ERROR /usr/local/lib/python3.10/site-packages/apache_beam/runners/worker/bundle_processor.py:237 [] - Forbidden [
while running 'Create|OpenURLWithFSSpec|OpenWithKerchunk|WriteCombinedReference/OpenURLWithFSSpec/MapWithConcurrencyLimit/open_url']     
Traceback (most recent call last):                                                                                                       
  File "/opt/apache/beam-venv/beam-venv-worker-1-1/lib/python3.10/site-packages/s3fs/core.py", line 113, in _error_wrapper               
    return await func(*args, **kwargs)                                                                                                   
  File "/opt/apache/beam-venv/beam-venv-worker-1-1/lib/python3.10/site-packages/aiobotocore/client.py", line 408, in _make_api_call      
    raise error_class(parsed_response, operation_name)                                                                                   
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden                                

The above exception was the direct cause of the following exception: