search

google-research / weatherbench2

A benchmark for the next generation of data-driven global weather models.
https://weatherbench2.readthedocs.io
Apache License 2.0
404 stars 41 forks source link

The production of the low resolution prediction results #174

Open rs-lsl opened 1 month ago

rs-lsl commented 1 month ago

Hello, I'm extremely grateful for your work. I'd like to clarify if the low-resolution prediction results, such as the one from Fuxi method named "2020-64x32_equiangular_conservative.zarr/", are obtained directly through training and prediction on a low-resolution dataset, or if they are downsampled from the original high-resolution prediction results. If they are obtained by downsampling, could you please elaborate on the specific method used for the downsampling process? Thank you very much.

raspstephan commented 1 month ago

Hi, the lower resolution results were regridded using: https://github.com/google-research/weatherbench2/blob/main/scripts/regrid.py with conservative regridding.

rs-lsl commented 1 month ago

Thank you ^_^

rs-lsl commented 1 month ago

Hello, when i use the https://github.com/google-research/weatherbench2/blob/main/scripts/regrid.py, i encounter with the following error, do you know how to solve this bug? Does this error is induced by the incorrect path? Looking forward your suggestions. Very very thanks.

I0801 10:38:46.446722 140473376526848 stager.py:761] Executing command: ['/home/lisl/.conda/envs/py39gdal/bin/python', '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmpkdxo0tlz/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp39', '--platform', 'manylinux2014_x86_64'] I0801 10:38:47.240728 140473376526848 stager.py:786] Executing command: ['/home/lisl/.conda/envs/py39gdal/bin/python', '-m', 'build', '--sdist', '--outdir', '/tmp/tmpzsqqg563', '/home/lisl/weatherbench2_main/scripts'] /home/lisl/.conda/envs/py39gdal/bin/python: No module named build I0801 10:38:47.282829 140473376526848 stager.py:796] Executing command: ['/home/lisl/.conda/envs/py39gdal/bin/python', 'setup.py', 'sdist', '--dist-dir', '/tmp/tmpzsqqg563'] I0801 10:38:47.477443 140473376526848 dataflow_runner.py:395] Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild I0801 10:38:47.481861 140473376526848 environments.py:314] Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/beam_python3.9_sdk:2.53.0 I0801 10:38:47.482055 140473376526848 environments.py:321] Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/beam_python3.9_sdk:2.53.0" for Docker environment I0801 10:38:48.289190 140473376526848 translations.py:712] ==================== <function pack_combiners at 0x7fc27af97310> ==================== I0801 10:38:48.289617 140473376526848 translations.py:712] ==================== <function sort_stages at 0x7fc27af97af0> ==================== I0801 10:38:48.294850 140473376526848 auth.py:134] Setting socket default timeout to 60 seconds. I0801 10:38:48.294947 140473376526848 auth.py:136] socket default timeout is 60.0 seconds. W0801 10:38:51.300006 140473376526848 _metadata.py:139] Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out W0801 10:38:51.375129 140473376526848 _metadata.py:139] Compute Engine Metadata server unavailable on attempt 2 of 3. Reason: [Errno 113] No route to host W0801 10:38:54.379387 140473376526848 _metadata.py:139] Compute Engine Metadata server unavailable on attempt 3 of 3. Reason: timed out W0801 10:38:54.379814 140473376526848 _default.py:338] Authentication failed using Compute Engine authentication due to unavailable metadata server. W0801 10:38:54.381045 140473376526848 retry.py:290] Retry with exponential backoff: waiting for 1.6381955742278063 seconds before retrying _get_credentials_with_retrys because we caught exception: google.auth.exceptions.DefaultCredentialsError: Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc for more information. Traceback for above exception (most recent call last): File "/home/lisl/.conda/envs/py39gdal/lib/python3.9/site-packages/apache_beam/utils/retry.py", line 275, in wrapper return fun(*args, **kwargs) File "/home/lisl/.conda/envs/py39gdal/lib/python3.9/site-packages/apache_beam/internal/gcp/auth.py", line 175, in _get_credentials_withretrys credentials, = google.auth.default( File "/home/lisl/.conda/envs/py39gdal/lib/python3.9/site-packages/google/auth/_default.py", line 691, in default raise exceptions.DefaultCredentialsError(_CLOUD_SDK_MISSING_CREDENTIALS)

W0801 10:38:59.027251 140473376526848 _metadata.py:139] Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out W0801 10:38:59.087109 140473376526848 _metadata.py:139] Compute Engine Metadata server unavailable on attempt 2 of 3. Reason: [Errno 113] No route to host W0801 10:39:02.090723 140473376526848 _metadata.py:139] Compute Engine Metadata server unavailable on attempt 3 of 3. Reason: timed out W0801 10:39:02.091118 140473376526848 _default.py:338] Authentication failed using Compute Engine authentication due to unavailable metadata server. W0801 10:39:02.091533 140473376526848 retry.py:290] Retry with exponential backoff: waiting for 3.814032520849619 seconds before retrying _get_credentials_with_retrys because we caught exception: google.auth.exceptions.DefaultCredentialsError: Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc for more information. Traceback for above exception (most recent call last): File "/home/lisl/.conda/envs/py39gdal/lib/python3.9/site-packages/apache_beam/utils/retry.py", line 275, in wrapper return fun(*args, **kwargs) File "/home/lisl/.conda/envs/py39gdal/lib/python3.9/site-packages/apache_beam/internal/gcp/auth.py", line 175, in _get_credentials_withretrys credentials, = google.auth.default( File "/home/lisl/.conda/envs/py39gdal/lib/python3.9/site-packages/google/auth/_default.py", line 691, in default raise exceptions.DefaultCredentialsError(_CLOUD_SDK_MISSING_CREDENTIALS)

W0801 10:39:08.915890 140473376526848 _metadata.py:139] Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out W0801 10:39:08.975111 140473376526848 _metadata.py:139] Compute Engine Metadata server unavailable on attempt 2 of 3. Reason: [Errno 113] No route to host W0801 10:39:11.978773 140473376526848 _metadata.py:139] Compute Engine Metadata server unavailable on attempt 3 of 3. Reason: timed out W0801 10:39:11.979199 140473376526848 _default.py:338] Authentication failed using Compute Engine authentication due to unavailable metadata server. W0801 10:39:11.979631 140473376526848 retry.py:290] Retry with exponential backoff: waiting for 4.959881430377898 seconds before retrying _get_credentials_with_retrys because we caught exception: google.auth.exceptions.DefaultCredentialsError: Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc for more information. Traceback for above exception (most recent call last): File "/home/lisl/.conda/envs/py39gdal/lib/python3.9/site-packages/apache_beam/utils/retry.py", line 275, in wrapper return fun(*args, **kwargs) File "/home/lisl/.conda/envs/py39gdal/lib/python3.9/site-packages/apache_beam/internal/gcp/auth.py", line 175, in _get_credentials_withretrys credentials, = google.auth.default( File "/home/lisl/.conda/envs/py39gdal/lib/python3.9/site-packages/google/auth/_default.py", line 691, in default raise exceptions.DefaultCredentialsError(_CLOUD_SDK_MISSING_CREDENTIALS)

W0801 10:39:19.950947 140473376526848 _metadata.py:139] Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out W0801 10:39:20.015117 140473376526848 _metadata.py:139] Compute Engine Metadata server unavailable on attempt 2 of 3. Reason: [Errno 113] No route to host W0801 10:39:23.018795 140473376526848 _metadata.py:139] Compute Engine Metadata server unavailable on attempt 3 of 3. Reason: timed out W0801 10:39:23.019207 140473376526848 _default.py:338] Authentication failed using Compute Engine authentication due to unavailable metadata server. W0801 10:39:23.019629 140473376526848 retry.py:290] Retry with exponential backoff: waiting for 11.64508406264211 seconds before retrying _get_credentials_with_retrys because we caught exception: google.auth.exceptions.DefaultCredentialsError: Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc for more information. Traceback for above exception (most recent call last): File "/home/lisl/.conda/envs/py39gdal/lib/python3.9/site-packages/apache_beam/utils/retry.py", line 275, in wrapper return fun(*args, **kwargs) File "/home/lisl/.conda/envs/py39gdal/lib/python3.9/site-packages/apache_beam/internal/gcp/auth.py", line 175, in _get_credentials_withretrys credentials, = google.auth.default( File "/home/lisl/.conda/envs/py39gdal/lib/python3.9/site-packages/google/auth/_default.py", line 691, in default raise exceptions.DefaultCredentialsError(_CLOUD_SDK_MISSING_CREDENTIALS)

W0801 10:39:37.682912 140473376526848 _metadata.py:139] Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out W0801 10:39:37.743090 140473376526848 _metadata.py:139] Compute Engine Metadata server unavailable on attempt 2 of 3. Reason: [Errno 113] No route to host W0801 10:39:40.746725 140473376526848 _metadata.py:139] Compute Engine Metadata server unavailable on attempt 3 of 3. Reason: timed out W0801 10:39:40.747148 140473376526848 _default.py:338] Authentication failed using Compute Engine authentication due to unavailable metadata server. W0801 10:39:40.747362 140473376526848 auth.py:165] Unable to find default credentials to use: Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc for more information. Connecting anonymously. This is expected if no credentials are needed to access GCP resources. Traceback (most recent call last): File "/home/lisl/weatherbench2_main/model2023/datasets/regrid.py", line 166, in app.run(main) File "/home/lisl/.conda/envs/py39gdal/lib/python3.9/site-packages/absl/app.py", line 308, in run _run_main(main, args) File "/home/lisl/.conda/envs/py39gdal/lib/python3.9/site-packages/absl/app.py", line 254, in _run_main sys.exit(main(argv)) File "/home/lisl/weatherbench2main/model2023/datasets/regrid.py", line 145, in main = ( File "/home/lisl/.conda/envs/py39gdal/lib/python3.9/site-packages/apache_beam/pipeline.py", line 612, in exit self.result = self.run() File "/home/lisl/.conda/envs/py39gdal/lib/python3.9/site-packages/apache_beam/pipeline.py", line 559, in run return Pipeline.from_runner_api( File "/home/lisl/.conda/envs/py39gdal/lib/python3.9/site-packages/apache_beam/pipeline.py", line 586, in run return self.runner.run_pipeline(self, self._options) File "/home/lisl/.conda/envs/py39gdal/lib/python3.9/site-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 477, in run_pipeline self.dataflow_client = apiclient.DataflowApplicationClient( File "/home/lisl/.conda/envs/py39gdal/lib/python3.9/site-packages/apache_beam/runners/dataflow/internal/apiclient.py", line 502, in init storage_credentials = credentials.get_google_auth_credentials() AttributeError: 'NoneType' object has no attribute 'get_google_auth_credentials'