search

lhenneman / hyspdisp

4 stars 3 forks source link

Data in the package. Selecting Units #25

Closed schoolAccountMajaG closed 4 years ago

schoolAccountMajaG commented 5 years ago

Data for units are also available for 2011 but the vignette does not mention it. Many of the units do not have stack height data. Why? Is it feasible for us to fill these missing values?

schoolAccountMajaG commented 5 years ago

https://github.com/lhenneman/hyspdisp/blob/master/vignettes/hyads.Rmd#L124-L134

schoolAccountMajaG commented 5 years ago

@cchoirat @lhenneman I would like to make a vignette on how this was prepared for the package. Do you have any R script?

cchoirat commented 5 years ago

More context would help :)

On Fri, Jun 28, 2019 at 1:42 PM Maja notifications@github.com wrote:

@cchoirat https://github.com/cchoirat @lhenneman https://github.com/lhenneman I would like to make a vignette on how this was prepared. Do you have any R script?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lhenneman/hyspdisp/issues/25?email_source=notifications&email_token=AA73AZIS2SMDDAPA3Z3Y2YLP4X2JRA5CNFSM4HXISIDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYZ3EDQ#issuecomment-506704398, or mute the thread https://github.com/notifications/unsubscribe-auth/AA73AZKQ4HUQP4VE3KZQSWTP4X2JRANCNFSM4HXISIDA .

schoolAccountMajaG commented 5 years ago

Also,

Many of the units in the provided datasets do not have stack height data. In these cases, it is suggested in Henneman et al. (2019) to fill with the average stack height of all units.

https://github.com/lhenneman/hyspdisp/blob/master/vignettes/hyads.Rmd#L124-L134

Can we do fill this and get a binary flag variables specifying which ones were filled?

schoolAccountMajaG commented 5 years ago

More context would help :) … On Fri, Jun 28, 2019 at 1:42 PM Maja @.***> wrote: @cchoirat https://github.com/cchoirat @lhenneman https://github.com/lhenneman I would like to make a vignette on how this was prepared. Do you have any R script? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#25?email_source=notifications&email_token=AA73AZIS2SMDDAPA3Z3Y2YLP4X2JRA5CNFSM4HXISIDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYZ3EDQ#issuecomment-506704398>, or mute the thread https://github.com/notifications/unsubscribe-auth/AA73AZKQ4HUQP4VE3KZQSWTP4X2JRANCNFSM4HXISIDA .

I mean. The vignette links to these websites https://ampd.epa.gov/ampd/ https://www.eia.gov/electricity/data/eia860/. What was done after accessing these pages to get the final units data that is in the package?

lhenneman commented 5 years ago

Yeah, I think this is a great idea.

@cchorat downloaded the data and wrote the code to do create the input files. For now, the code is in 2 scripts that need to be run back-to-back. Here are the two scripts for 2006, for example: https://www.dropbox.com/s/c91fa8rkqyf58le/merge_ampd_nei_coal_only_2006.R?dl=0 https://www.dropbox.com/s/zweas0l2qeip7wd/create_final_nei_ampd_merge_all_units_2006.R?dl=0

The first script takes as input both the AMPD data and NEI data: https://www.dropbox.com/s/5qorstjut36rup6/AMPD_Unit_with_Sulfur_Content_and_Regulations_with_Facility_Attributes.csv?dl=0 https://www.dropbox.com/s/u4l3oej7j5egwed/ptegu_2014NEIv1_final_POINT_02nov2016_v0.csv?dl=0

Relatedly, the AMPD data that goes into the merge script above is the same as the "PP.units.monthly1995_2017" dataset that's currently included in the package. I create that file using this code: https://www.dropbox.com/s/fkx83cn10qfwuog/edit_AMPDdata.R?dl=0

This script uses the AMPD_Unit.csv file, which is a slimmed-down version of the AMPD file above: https://www.dropbox.com/s/gv81qpc5gm00f0s/AMPD_Unit.csv?dl=0

You're pointing out good issues here...none of this was done in a great way. It'd be great if this could all be documented well and streamlined.

On Fri, Jun 28, 2019 at 7:47 AM Maja notifications@github.com wrote:

More context would help :) … <#m4447545757922300485> On Fri, Jun 28, 2019 at 1:42 PM Maja @.***> wrote: @cchoirat https://github.com/cchoirat https://github.com/cchoirat @lhenneman https://github.com/lhenneman https://github.com/lhenneman I would like to make a vignette on how this was prepared. Do you have any R script? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#25 https://github.com/lhenneman/hyspdisp/issues/25?email_source=notifications&email_token=AA73AZIS2SMDDAPA3Z3Y2YLP4X2JRA5CNFSM4HXISIDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYZ3EDQ#issuecomment-506704398>, or mute the thread https://github.com/notifications/unsubscribe-auth/AA73AZKQ4HUQP4VE3KZQSWTP4X2JRANCNFSM4HXISIDA .

I mean. The vignette links to these packages https://ampd.epa.gov/ampd/ https://www.eia.gov/electricity/data/eia860/. What was done after accessing these pages to get the final units data that is in the package.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lhenneman/hyspdisp/issues/25?email_source=notifications&email_token=AHMZYW5NM2ODPZ3Y2NZRXL3P4X23ZA5CNFSM4HXISIDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYZ3NFY#issuecomment-506705559, or mute the thread https://github.com/notifications/unsubscribe-auth/AHMZYW6ABZBGTAQJWMUHWBTP4X23ZANCNFSM4HXISIDA .

-- Lucas R.F. Henneman, Ph.D. (404) 788-2161 lhenneman@gmail.com

schoolAccountMajaG commented 5 years ago

https://www.dropbox.com/s/u4l3oej7j5egwed/ptegu_2014NEIv1_final_POINT_02nov2016_v0.csv?dl=0

https://www.epa.gov/air-emissions-inventories/2014-national-emissions-inventory-nei-data @cchoirat , @lhenneman do you know what exactly was downloaded from here?

cchoirat commented 5 years ago

There are R scripts if I remember correctly.

On Mon, Jul 1, 2019 at 11:23 AM Maja notifications@github.com wrote:

https://www.dropbox.com/s/u4l3oej7j5egwed/ptegu_2014NEIv1_final_POINT_02nov2016_v0.csv?dl=0

https://www.epa.gov/air-emissions-inventories/2014-national-emissions-inventory-nei-data @cchoirat https://github.com/cchoirat , @lhenneman https://github.com/lhenneman do you know what exactly was downloaded from here?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lhenneman/hyspdisp/issues/25?email_source=notifications&email_token=AA73AZJTBX6B2EUKMWZCANLP5HEIPA5CNFSM4HXISIDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY5RD4Q#issuecomment-507187698, or mute the thread https://github.com/notifications/unsubscribe-auth/AA73AZLI62L5HXHZMTAWQ6TP5HEIPANCNFSM4HXISIDA .

schoolAccountMajaG commented 5 years ago

Yes, but they are using a file that is on a dropbox. How can the user find it if they want to reproduce?

cchoirat commented 5 years ago

I mean I don't have the folder in sync on my machine. But @lhenneman can you please share the DB?

On Mon, Jul 1, 2019 at 1:03 PM Maja notifications@github.com wrote:

Yes, but they are using a file that is on a dropbox. How can the user find it if they want to reproduce?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lhenneman/hyspdisp/issues/25?email_source=notifications&email_token=AA73AZJFC76DVP5FT3RUDI3P5HQBJA5CNFSM4HXISIDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY5YYBI#issuecomment-507218949, or mute the thread https://github.com/notifications/unsubscribe-auth/AA73AZLOPBFIQNNMUPM7AJ3P5HQBJANCNFSM4HXISIDA .

schoolAccountMajaG commented 5 years ago

I am sorry to be confusing. I do have the files. I am creating a vignette that explains the whole data manipulation starting from the original data if someone else wanted to access them to replicate the whole process. I think it will be difficult to easily reproduce the data manipulation process unless we share the dropbox file or provide specific instruction on how to download the exact same file from the web.

It is hard to find out which file exactly was downloaded from https://www.epa.gov/air-emissions-inventories/2014-national-emissions-inventory-nei-data. Maybe because I do not have enough background knowledge.

cchoirat commented 5 years ago

I mean there should be more info on DB. Do you already have access to the folder?

On Mon, Jul 1, 2019 at 1:30 PM Maja notifications@github.com wrote:

I am sorry to be confusing. I do have the files. I am creating a vignette that explains the whole data manipulation starting from the original data if someone else wanted to access them to replicate the whole process. I think it will be difficult to easily reproduce the data manipulation process unless we share the dropbox file or provide specific instruction on how to download the exact same file from the web.

It is hard to find out which file exactly was downloaded from https://www.epa.gov/air-emissions-inventories/2014-national-emissions-inventory-nei-data. Maybe because I do not have enough background knowledge.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lhenneman/hyspdisp/issues/25?email_source=notifications&email_token=AA73AZOMYTRQMSWSLHNY6Z3P5HTF7A5CNFSM4HXISIDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY52TUA#issuecomment-507226576, or mute the thread https://github.com/notifications/unsubscribe-auth/AA73AZPNWJOWB6LQAJAZNYTP5HTF7ANCNFSM4HXISIDA .

schoolAccountMajaG commented 5 years ago

I do not think so.

lhenneman commented 5 years ago

Maja, I think the files I sent are all the files that have info on this, but here's a link the entire DB directory: https://www.dropbox.com/sh/02uwlhvwkvwg7p3/AADffEkftUMW3OASCsuOBXzQa?dl=0.

I'm not sure exactly how the data was downloaded, and I can't seem to find it on the NEI website. @cchoirat, do you have any memory of how you got the 2014fa_nata_cb6cmaq_14j directory?

On Mon, Jul 1, 2019 at 7:39 AM Maja notifications@github.com wrote:

I do not think so.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lhenneman/hyspdisp/issues/25?email_source=notifications&email_token=AHMZYWYULW63WLBEZKCTLCDP5HUHPA5CNFSM4HXISIDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY53H7Q#issuecomment-507229182, or mute the thread https://github.com/notifications/unsubscribe-auth/AHMZYW42WUKYOY3LWMX34Q3P5HUHPANCNFSM4HXISIDA .

-- Lucas R.F. Henneman, Ph.D. (404) 788-2161 lhenneman@gmail.com

lhenneman commented 5 years ago

Maybe here - ftp://newftp.epa.gov/air/emismod/2014/v2/2014fd/emissions/

Try the 2014fd_inputs_point.zip file. Within that, the 2014fd_cb6_14j/inputs/ptegu/ptegu_2014NEIv2_POINT_20171103_final_21dec2017_nf_v2.csv folder should correspond to what we're using.

On Mon, Jul 1, 2019 at 8:52 AM Lucas Henneman lhenneman@gmail.com wrote:

Maja, I think the files I sent are all the files that have info on this, but here's a link the entire DB directory: https://www.dropbox.com/sh/02uwlhvwkvwg7p3/AADffEkftUMW3OASCsuOBXzQa?dl=0 .

I'm not sure exactly how the data was downloaded, and I can't seem to find it on the NEI website. @cchoirat, do you have any memory of how you got the 2014fa_nata_cb6cmaq_14j directory?

On Mon, Jul 1, 2019 at 7:39 AM Maja notifications@github.com wrote:

I do not think so.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lhenneman/hyspdisp/issues/25?email_source=notifications&email_token=AHMZYWYULW63WLBEZKCTLCDP5HUHPA5CNFSM4HXISIDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY53H7Q#issuecomment-507229182, or mute the thread https://github.com/notifications/unsubscribe-auth/AHMZYW42WUKYOY3LWMX34Q3P5HUHPANCNFSM4HXISIDA .

-- Lucas R.F. Henneman, Ph.D. (404) 788-2161 lhenneman@gmail.com

-- Lucas R.F. Henneman, Ph.D. (404) 788-2161 lhenneman@gmail.com

schoolAccountMajaG commented 4 years ago

"Vignette - Units Data Preparation" shows all the steps