laestrada
commented
1 year ago Hi @psadavarte,
Thanks for writing in -- I hope you are finding the IMI to be useful!
For your poland inversion it seems that the paths for your simulations are mixed between .25 and .5 degree grids. Eg. in your log file it tries to download GEOS_0.25x0.3125/GEOS_FP/2018/05/GEOSFP.20180502.A1.05x0625.EU.nc. This is a known bug with using the .5 degree option for the IMI and is fixed in the development branch. This fix will be included in IMI 1.2, which we are planning to release in the next 2 weeks. I would recommend sticking with the .25 degree resolution until then.
For your inversion over South Africa, for now .5 degree resolution is not supported over African domains. This is due to storage space constraints for hosting the meteorological files driving the simulations, so we are prioritizing the higher resolution option.
If you are worried about the increased computational expense for using the .25 degree option, IMI 1.2 will also include options to reduce the state vector dimension, reducing the computational expense by effectively reducing resolution over areas with low emissions and maintaining high resolution where there are strong emissions.
Let us know if this resolves your issue (or if you run into any additional problems) so that I can close the issue.
Best, Lucas and the IMI Team
psadavarte
Hi IMI support team
What institution are you from?
I'm a Research Scientist from Duke University.
Description of the problem
I'm a beginner-level researcher in terms of using inversion for my study. Currently, I'm running the IMI on AWS and following the steps provided on the website https://imi.readthedocs.io/en/latest/getting-started/quick-start.html I was able to generate the results over Permian for 1-week which come pre-loaded. For my study I tried replicating the same, but over different regions. Initially, I started with my focus on Poland and changing the meteorology and resolution.
Given that, I modified the config.yml file (copied at the bottom) and provided the lat-lon for my domain study, along with met changed to "merra2" and res changed to "0.5x0.625". While retaining the rest of the setting as it is. The idea was to gradually modify the configuration file as per the study. I've also attached the respective log file to understand the error I received. Please take a look.
Apart from just modifying the config.yml files, do I have to modify and execute any other file before running the imi executable?
Description of troubleshooting performed
I did some troubleshooting here. For eg, I changed the region (changed the lat-lon) while keeping the rest of the configuration as it is, which gave me a completely different error. Log files are attached for your perusal.
IMI version
Latest Version: imi-1.1.0
Attach relevant files
Copied relevant files
Thanking you in advance. I hope I have provided sufficient information. imi_output_southafrica.log imi_output_poland.log
=============== Config_poland.yml==================
General
RunName: "Test_Poland_1week" isAWS: true UseSlurm: true SafeMode: true
Period of interest
StartDate: 20180501 EndDate: 20180508 SpinupMonths: 1
Region of interest
These lat/lon bounds are only used if CreateAutomaticRectilinearStateVectorFile: true
Otherwise lat/lon bounds are determined from StateVectorFile
LonMin: 18.40 LonMax: 19.40 LatMin: 49.50 LatMax: 50.50
Use nested grid simulation?
Must be "true" for IMI regional inversions
NestedGrid: true
Select nested grid region (for using pre-cropped meteorological fields)
Current options are listed below with ([lat],[lon]) bounds:
"AF" : Africa ([-37,40], [-20,53])
"AS" : Asia ([-11,55],[60,150])
"EU" : Europe ([33,61],[-30,70])
"ME" : Middle East ([12,44], [-20,70])
"NA" : North America ([10,70],[-140,-40])
"OC" : Oceania ([-50,5], [110,180])
"RU" : Russia ([41,83], [19,180])
"SA" : South America ([-59,16], [-88,-31])
"" : Use for global met fields (global simulation/custom nested grids)
For example, if the region of interest is in Europe ([33,61],[-30,70]), select "EU".
NestedRegion: "EU"
State vector
CreateAutomaticRectilinearStateVectorFile: true nBufferClusters: 8 BufferDeg: 5 LandThreshold: 0.25
Custom state vector
StateVectorFile: "/home/ubuntu/integrated_methane_inversion/resources/statevectors/StateVector.nc" ShapeFile: "/home/ubuntu/integrated_methane_inversion/resources/shapefiles/PermianBasin_Extent_201712.shp"
Inversion
PriorError: 0.5 ObsError: 15 Gamma: 1.0 PrecomputedJacobian: false
Grid
Select "0.25x0.3125" and "geosfp", or "0.5x0.625" and "merra2"
Res: "0.5x0.625" Met: "merra2"
Setup modules
Turn on/off different steps in setting up the inversion
SetupTemplateRundir: true SetupSpinupRun: false SetupJacobianRuns: false SetupInversion: false SetupPosteriorRun: false
Run modules
Turn on/off different steps in performing the inversion
RunSetup: true DoSpinup: false DoJacobian: false DoInversion: false DoPosterior: false
IMI preview
DoPreview: true DOFSThreshold: 0
====================================================================
Advanced Settings (optional)
====================================================================
These settings are intended for advanced users who wish to:
a. modify additional GEOS-Chem options, or
b. run the IMI on a local cluster.
They can be ignored for any standard cloud application of the IMI.
--------------------------------------------------------------------
Additional settings for GEOS-Chem simulations
--------------------------------------------------------------------
Jacobian settings
PerturbValue: 1.5
Apply scale factors from a previous inversion?
UseEmisSF: false UseOHSF: false
Save out hourly diagnostics from GEOS-Chem?
For use in satellite operators via post-processing -- required for TROPOMI
inversions
HourlyCH4: true
Turn on planeflight diagnostic in GEOS-Chem?
For use in comparing GEOS-Chem against planeflight data. The path
to those data must be specified in input.geos.
PLANEFLIGHT: false
Turn on old observation operators in GEOS-Chem?
These will save out text files comparing GEOS-Chem to observations, but have
to be manually incorporated into the IMI
GOSAT: false TCCON: false AIRS: false
--------------------------------------------------------------------
Settings for running on a local cluster
--------------------------------------------------------------------
Path for IMI runs and output
OutputPath: "/home/ubuntu/imi_output_dir"
Path to GEOS-Chem input data
DataPath: "/home/ubuntu/ExtData"
Environment files
CondaFile: "/home/ubuntu/miniconda/etc/profile.d/conda.sh" CondaEnv: "geo"
Download initial restart file from AWS S3?
RestartDownload: true
Path to initial GEOS-Chem restart file + prefix
("YYYYMMDD_0000z.nc4" will be appended)
RestartFilePrefix: "/home/ubuntu/ExtData/BoundaryConditions/GEOSChem.BoundaryConditions." RestartFilePreviewPrefix: "/home/ubuntu/ExtData/BoundaryConditions/GEOSChem.BoundaryConditions."
Path to GEOS-Chem boundary condition files (for nested grid simulations)
BCpath: "/home/ubuntu/ExtData/BoundaryConditions"
Options to download missing GEOS-Chem input data from AWS S3
NOTE: You will be charged if your ec2 instance is not in the
us-east-1 region.
PreviewDryRun: true SpinupDryrun: true ProductionDryRun: true PosteriorDryRun: true BCdryrun: true
=============== Config_southafrica.yml==================
General
RunName: "Test_Southafrica_1week" isAWS: true UseSlurm: true SafeMode: true
Period of interest
StartDate: 20180501 EndDate: 20180508 SpinupMonths: 1
Region of interest
These lat/lon bounds are only used if CreateAutomaticRectilinearStateVectorFile: true
Otherwise lat/lon bounds are determined from StateVectorFile
LonMin: 27.40 LonMax: 28.60 LatMin: -13.10 LatMax: -12.10
Use nested grid simulation?
Must be "true" for IMI regional inversions
NestedGrid: true
Select nested grid region (for using pre-cropped meteorological fields)
Current options are listed below with ([lat],[lon]) bounds:
"AF" : Africa ([-37,40], [-20,53])
"AS" : Asia ([-11,55],[60,150])
"EU" : Europe ([33,61],[-30,70])
"ME" : Middle East ([12,44], [-20,70])
"NA" : North America ([10,70],[-140,-40])
"OC" : Oceania ([-50,5], [110,180])
"RU" : Russia ([41,83], [19,180])
"SA" : South America ([-59,16], [-88,-31])
"" : Use for global met fields (global simulation/custom nested grids)
For example, if the region of interest is in Europe ([33,61],[-30,70]), select "EU".
NestedRegion: "AF"
State vector
CreateAutomaticRectilinearStateVectorFile: true nBufferClusters: 8 BufferDeg: 5 LandThreshold: 0.25
Custom state vector
StateVectorFile: "/home/ubuntu/integrated_methane_inversion/resources/statevectors/StateVector.nc" ShapeFile: "/home/ubuntu/integrated_methane_inversion/resources/shapefiles/PermianBasin_Extent_201712.shp"
Inversion
PriorError: 0.5 ObsError: 15 Gamma: 1.0 PrecomputedJacobian: false
Grid
Select "0.25x0.3125" and "geosfp", or "0.5x0.625" and "merra2"
Res: "0.5x0.625" Met: "merra2"
Setup modules
Turn on/off different steps in setting up the inversion
SetupTemplateRundir: true SetupSpinupRun: false SetupJacobianRuns: false SetupInversion: false SetupPosteriorRun: false
Run modules
Turn on/off different steps in performing the inversion
RunSetup: true DoSpinup: false DoJacobian: false DoInversion: false DoPosterior: false
IMI preview
DoPreview: true DOFSThreshold: 0
====================================================================
Advanced Settings (optional)
====================================================================
These settings are intended for advanced users who wish to:
a. modify additional GEOS-Chem options, or
b. run the IMI on a local cluster.
They can be ignored for any standard cloud application of the IMI.
--------------------------------------------------------------------
Additional settings for GEOS-Chem simulations
--------------------------------------------------------------------
Jacobian settings
PerturbValue: 1.5
Apply scale factors from a previous inversion?
UseEmisSF: false UseOHSF: false
Save out hourly diagnostics from GEOS-Chem?
For use in satellite operators via post-processing -- required for TROPOMI
inversions
HourlyCH4: true
Turn on planeflight diagnostic in GEOS-Chem?
For use in comparing GEOS-Chem against planeflight data. The path
to those data must be specified in input.geos.
PLANEFLIGHT: false
Turn on old observation operators in GEOS-Chem?
These will save out text files comparing GEOS-Chem to observations, but have
to be manually incorporated into the IMI
GOSAT: false TCCON: false AIRS: false
--------------------------------------------------------------------
Settings for running on a local cluster
--------------------------------------------------------------------
Path for IMI runs and output
OutputPath: "/home/ubuntu/imi_output_dir"
Path to GEOS-Chem input data
DataPath: "/home/ubuntu/ExtData"
Environment files
CondaFile: "/home/ubuntu/miniconda/etc/profile.d/conda.sh" CondaEnv: "geo"
Download initial restart file from AWS S3?
RestartDownload: true
Path to initial GEOS-Chem restart file + prefix
("YYYYMMDD_0000z.nc4" will be appended)
RestartFilePrefix: "/home/ubuntu/ExtData/BoundaryConditions/GEOSChem.BoundaryConditions." RestartFilePreviewPrefix: "/home/ubuntu/ExtData/BoundaryConditions/GEOSChem.BoundaryConditions."
Path to GEOS-Chem boundary condition files (for nested grid simulations)
BCpath: "/home/ubuntu/ExtData/BoundaryConditions"
Options to download missing GEOS-Chem input data from AWS S3
NOTE: You will be charged if your ec2 instance is not in the
us-east-1 region.
PreviewDryRun: true SpinupDryrun: true ProductionDryRun: true PosteriorDryRun: true BCdryrun: true