module G issue

[biancaglez]

I have cloned the repo and now have access to all folders in CEDS, and am running Module G before I edit to extract data points (instead of gridded values).

The fourth line of code: initialize("G1.1.grid_bulk_emissions.R", log_msg, headers)) gives the following error:

initialize( "G1.1.grid_bulk_emissions.R", log_msg, headers )initialize( "G1.1.grid_bulk_emissions.R", log_msg, headers )Error in file(file, if (append) "a" else "w") : cannot open the connection In addition: Warning message: In file(file, if (append) "a" else "w") : cannot open file '../code/parameters//../../logs/G1.1.grid_bulk_emissions.R.log': No such file or directory Called from: file(file, if (append) "a" else "w")

@ssmithClimate has been my main point of contact and very helpful!

[ssmithClimate]

Do you have write permissions for the directories in the CEDS folder? Also, how are you running the system? From the command line?

[biancaglez]

I downloaded the zip instead of cloning it.

-- I'm running the system from the R console

[ssmithClimate]

Please see wiki instructions on running gridding:

did you also download the gridding proxies package?

[biancaglez]

@ssmithClimate -- do you mind giving me permissions to the CEDS folder? It seems as if downloading and unzipping (tried two different times now) results in only part of the data being downloaded. Not sure why if I wait for all of the files to unzip, but alas.

[ssmithClimate]

You should be able to fork CEDS and you will get all the files that are in CEDS. Note, however, not all the necessary gridding files are on CEDS, you separately need to download the gridding proxy files as described in the wiki instructions referenced above.

[biancaglez]

Hi @ssmithClimate

Thanks again for the above guidance.

I have followed the instructions in your above link:

Emissions by country, sector, fuel must first be generated by running the CEDS system.

In order to produce gridded emissions, the gridded proxy data must be obtained from zenodo through this link.

The package there contains four folders: mask, proxy, proxy-backup, seasonality

Copy these folders into the input/gridding folder in your CEDS directory. Assuming you do not have any previously modified CEDS proxy data files in your system, you can replace the folders that are already there from the GitHub distribution (you will note that those folders are otherwise empty in the CEDS GitHub distribution).

Gridded emissions can then be produced using the makefile system, for example make BC-gridded. Note that if the final emissions file is not up to date, the make BC-gridded command will re-run the entire system as needed.

Note: users should edit the netCDF metadata as instructed in the file:

code/parameters/nc_generation_functions.R

and then in command prompt ran Rscript C:/Users/GonzalezB2/Desktop/Smithsonian/CEDS/code/module-G/G1.1.grid_bulk_emissions.R BC --nosave --no-restore

and am getting the error:

Error in file(filename, "r", encoding = encoding) : cannot open the connection Calls: source -> file In addition: Warning message: In file(filename, "r", encoding = encoding) : cannot open file '../code/parameters/header.R': No such file or directory Execution halted

I also tried to run G1.1 script in R (line by line) and on line 68 or

target_filename <- list.files( final_emissions_dir, pattern )
target_filename <- tools::file_path_sans_ext( target_filename )
stopifnot( length( target_filename ) == 1 )

I get the following error:

> stopifnot( length( target_filename ) == 1 )
Error: length(target_filename) == 1 is not TRUE

Any additional guidance to get this file running before I begin editing is appreciated!

[ssmithClimate]

What are you using as your CEDS final emissions data and where is this placed? It looks like you don't have CEDS emissions available in the location the script is expecting.

[biancaglez]

I don't know where to find the CEDS final emissions data. Do I need to run another script to generate this prior to running Module G?

[ssmithClimate]

If you've purchased the IEA energy data then you would install that (see wiki users guide) and run the system using "make" commands.

You can also use one of the release versions of the emissions. I would recommend the latest release - there is a link to this on the home page of this repo (e.g., https://zenodo.org/record/3606753).

You'll see in the module G code (for example G1.1.grid_bulkemissions.R) that the code is set up to read emission files with the naming convention: `pattern <- paste0( ".*", em, '_emissions_by_country_CEDS_sector.*' ) located here: final_emissions_dir <- filePath( "FIN_OUT", "current-versions/", extension = "" ) where the FIN_OUT directory is defined as./CEDS/final-emissions`

You can change those locations as needed.

[biancaglez]

Okay. Thanks for your time -- once I get this figured out, I'm happy to write a little guide explaining how to do these particular steps for specific sites if it helps

I downloaded the emissions data from above and located in that folder-- thanks. For now, there are additional errors in this module G script (the data files are in the paths indicated in script)

or lines 75-81 in the script:

location_index             <- readData( 'GRIDDING', domain_extension = 'gridding_mappings/', 'country_location_index_05' )
ceds_gridding_mapping      <- readData( 'GRIDDING', domain_extension = 'gridding_mappings/', 'CEDS_sector_to_gridding_sector_mapping' )
proxy_mapping              <- readData( 'GRIDDING', domain_extension = 'gridding_mappings/', 'proxy_mapping' )
seasonality_mapping        <- readData( 'GRIDDING', domain_extension = 'gridding_mappings/', 'seasonality_mapping' )
proxy_substitution_mapping <- readData( 'GRIDDING', domain_extension = 'gridding_mappings/', 'proxy_subsititution_mapping' )
sector_name_mapping        <- readData( 'GRIDDING', domain_extension = 'gridding_mappings/', 'CEDS_gridding_sectors' )
sector_name_mapping        <- unique( sector_name_mapping[ , c( 'CEDS_fin_sector', 'CEDS_fin_sector_short' ) ] )

[ssmithClimate]

In a case like this what I suggest is running the script in RStudio, in which case you can look at the variables to more directly to see what the issue is.

Which data source did you use for the emissions data?

[biancaglez]

Hi Steve,

I am using the data from zenodo and in particular the file named: CEDS_BC_emissions_by_country_CEDS_sector_v_2019_12_23

The above error is generated when running the module G script in R Studio. Before getting the above ERROR however, I did get this warning:

Warning message: In readMetaData(meta_domain, file_name, extension, meta_domain_ext = domain_extension) : Metadata input file missing for data file CEDS_BC_emissions_by_country_CEDS_sector_v_2019_12_23. System is generating a general metadata note...

Could it be that a metadata file is needed to run the readData functions above? And if so, which of these files contains metadata? I tried pasting all the zenodo final emissions files in to the directory as well and that didn't seem to patch the problem....

[ssmithClimate]

My apologies, you found a bug in the data read-in code. While not having a metadata file shouldn't be an issue it seems to be now - perhaps due to a change in r package behavior. We'll work on that, but in the meantime, to fix that you can add this parameter: , meta = FALSE )

to the end of each of those readData commands to turn off its attempt to read metadata.

[biancaglez]

Great, thanks!

[biancaglez]

Hi @ssmithClimate! A question about the files generated from the G1 script.

In the g1.1 grid bulk emissions script, I am reading in the following file from the zendo link you've shared CEDS_SO2_emissions_by_country_CEDS_sector_v_2019_12_23

The output file from this script according to the script should be : Output Files: MEDOUT: ` CEDS[em]anthro[year]0.5[CEDS_version].nc **—>** & I can confirm these are being properly generated like so :CEDS_SO2_anthro_1750_0.5.nc`

When I read these output files as a raster brick -- there is only one category or title in the slot of this raster (in this case - agriculture).

I'm asking because 1. not all sectors are present when reading this generated NC file and 2. The agriculture data is all 0s for all of my sites of interest across all years.

Are the generated files from the G1 script all for agriculture? How do I generate the other sectors from the NC files? (or could it be the way that the NC file is being read?) I will continue searching but thought I'd put the question out there... 🦖

If it helps... below is my code to read in generated NC files

for(file in ncfilenames){
    print(file)
    ## extract emission values from raster object at forestgeo points
    r <- brick(paste0(getwd(),"/", file))  # create a raster brick object from .nc file
    x <- raster::extract(r,points) # extract values from raster-brick at pts

    ### add relevent columns
    x <- cbind(sites.sitename = gsub(" ", "_", points@data$Site.name), x) # add sites.sitename
    x <- cbind(emission_source = rep(r@title, nrow(x)), x)
    x <- cbind(emission_units = rep(r@data@unit, nrow(x)), x)
    x <- cbind(emission_type = rep(stringr::str_sub(file, start=6, end=8), nrow(x)),x)

    if(file[1]==ncfilenames[1]){
        all_DEP <- x
    }
    else {
        print("merging years")
        all_DEP <- merge(all_DEP, x, by = c("sites.sitename", "emission_source", "emission_type", "emission_units"), all = T)
    }}

[ssmithClimate]

I suggest opening the original netCDF file in Panoply (free netCDF reader) to verify that you're looking at the right part of the file.

Note that the agriculture sector is just agricultural non-combustion emissions. So SO2 will generally be zero as these originate from combustion processes. (See sector definitions in the Hoesly et al. paper.) Combustion emissions are aggregated to the IND, RCO, etc. sectors (depending on how these are represented in the fuel consumption data).

[biancaglez]

Great! I've got values at all of my sites. I only had to specify a varname when creating a raster brick in my for loop. Thanks for the guidance