search

RamiKrispin / ts-cluster-analysis-r

Materials for the the Analyzing Time Series at Scale with Cluster Analysis in R Workshop
https://ramikrispin.github.io/ts-cluster-analysis-r/
19 stars 4 forks source link

Cannot follow the actions taken in the tutorial using RStudio #1

Open Iowastater opened 1 month ago

Iowastater commented 1 month ago

Hello Rami, I took your tutorial last week, and am trying to follow along with the recording.

I learned R on base R 20 years ago. Now, I work with RStudio or base R. I am not able to follow along with your demo. See questions in the Word doc attached – they may not make sense, because I don’t know how to describe your application in the video. It looks like you built a website that runs R code?

I would appreciate it if you can describe where the R scripts are to run your video actions using base R or RStudio.

Thank you, Paul Prew

Rstudio can't run code.docx TS_cluster_inR.zip

RamiKrispin commented 1 month ago

Hi Paul,

I run my R code inside VScode using a Docker container to set the environment. You can run it with Rstudio, but you will need to set up the required environment. Here is the list of libraries I am using to set up the environment: https://github.com/RamiKrispin/ts-cluster-analysis-r/blob/main/docker/setting_files/packages.json

In addition, you will need to install Quarto. Once you have the environment set, here are the steps to create the data and features:

You can skip those steps and load the data directly from the data folder.

Hope this helps.

Iowastater commented 4 weeks ago

Hello Rami, thank you for your reply. I have one main problem at the moment – I cannot find “eia_metadata” in the zip file that you provided with the class. Can you please just mail me that file?

I have been using R for 20 years, but I am unfamiliar with some of the terms you are using. All my work uses only 3 R objects: data frames, packages, and scripts. I hope you can guide me as I try to decipher your email.

“I run my R code inside VScode using a Docker container to set the environment.”

Docker is blocked at my work as a security risk. I do not know what VScode is, so I can’t run that.

“You can run it with Rstudio, but you will need to set up the required environment. Here is the list of libraries I am using to set up the environment:” https://urldefense.com/v3/__https:/github.com/RamiKrispin/ts-cluster-analysis-r/blob/main/docker/setting_files/packages.json__;!!Nkc5UzxO!rofLxXR9peFl2ArGSK5Zo63tcJG6OR3xA7lmaADfc_str8YpN3UgeevtzUrk0LAxuqDAfO8b1yfSfThBpSqMeDeolQ$

I don’t know what an environment is, but I was able to install the list of packages.

library(dplyr), library(ggplot2), library(shiny), library(plotly), library(EIAapi), library(tsibble), library(tsfeatures), library(feasts)

“In addition, you will need to install Quarto. Once you have the environment set, here are the steps to create the data and features:

You can skip those steps and load the data directly from the data folder.”

I loaded Quarto. Below is the data that is in the data folder, which I can easily load into R. But I do not see “eia_metadaa”, and without that file I cannot run the analyses.

@.***

@.***

But I cannot use the data to make the plot.

Everything depends on the object “eia_metadata”, but it I cannot find it

@.***

[A screen shot of a graph Description automatically generated]

@.***

So we will proceed without the other software, only R.

From: Rami Krispin @.> Sent: Wednesday, October 23, 2024 3:55 PM To: RamiKrispin/ts-cluster-analysis-r @.> Cc: Prew, Paul @.>; Author @.> Subject: Re: [RamiKrispin/ts-cluster-analysis-r] Cannot follow the actions taken in the tutorial using RStudio (Issue #1)

Hi Paul, I run my R code inside VScode using a Docker container to set the environment. You can run it with Rstudio, but you will need to set up the required environment. Here is the list of libraries I am using to set up the environment: https: //github. com/RamiKrispin/ts-cluster-analysis-r/blob/main/docker/setting_files/packages. json ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd

Hi Paul,

I run my R code inside VScode using a Docker container to set the environment. You can run it with Rstudio, but you will need to set up the required environment. Here is the list of libraries I am using to set up the environment: https://github.com/RamiKrispin/ts-cluster-analysis-r/blob/main/docker/setting_files/packages.jsonhttps://urldefense.com/v3/__https:/github.com/RamiKrispin/ts-cluster-analysis-r/blob/main/docker/setting_files/packages.json__;!!Nkc5UzxO!rofLxXR9peFl2ArGSK5Zo63tcJG6OR3xA7lmaADfc_str8YpN3UgeevtzUrk0LAxuqDAfO8b1yfSfThBpSqMeDeolQ$

In addition, you will need to install Quarto. Once you have the environment set, here are the steps to create the data and features:

You can skip those steps and load the data directly from the data folder.

Hope this helps.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/RamiKrispin/ts-cluster-analysis-r/issues/1*issuecomment-2433446778__;Iw!!Nkc5UzxO!rofLxXR9peFl2ArGSK5Zo63tcJG6OR3xA7lmaADfc_str8YpN3UgeevtzUrk0LAxuqDAfO8b1yfSfThBpSpoFBdqhA$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AB3IZYDJ7X2PSMRRD23227LZ5AELVAVCNFSM6AAAAABQPTUBU2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZTGQ2DMNZXHA__;!!Nkc5UzxO!rofLxXR9peFl2ArGSK5Zo63tcJG6OR3xA7lmaADfc_str8YpN3UgeevtzUrk0LAxuqDAfO8b1yfSfThBpSoIJsnIrw$. You are receiving this because you authored the thread.Message ID: @.**@.>>

CONFIDENTIALITY NOTICE: This e-mail communication and any attachments may contain proprietary and privileged information for the use of the designated recipients named above. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

RamiKrispin commented 4 weeks ago

All the data used in the workshop should be stored in the repository data folder. The error you are getting is related to the eia_metadata function from the EIAapi library. Here is how it is being used in the notebook:

meta_gas_raw <- eia_metadata(
    api_path = "natural-gas/cons/sum/data/facet/parent",
    api_key = api_key
)

Please make sure that you register to the API and you have a valid API key and assign it to the api_key variable.

Iowastater commented 4 weeks ago

Hello Ramik, I got the key for the API, and tried to use it within the R code that was in the HTML document.

api_key <- Sys.getenv("LxL4jhAqs3LJvoZ0HMPd0l0OXb4g3jocDwcUCg0z") api_path_gas <- "natural-gas/cons/sum/data/" meta_gas_raw <- eia_metadata(

  • api_path = "natural-gas/cons/sum/data/facet/parent",
  • api_key = api_key
  • ) Error in eia_metadata(api_path = "natural-gas/cons/sum/data/facet/parent", : could not find function "eia_metadata"

I also tried running the key without the quotation marks around it

api_key <- Sys.getenv(LxL4jhAqs3LJvoZ0HMPd0l0OXb4g3jocDwcUCg0z) Error: object 'LxL4jhAqs3LJvoZ0HMPd0l0OXb4g3jocDwcUCg0z' not found

Please advise. I am interested to be able to reproduce the R R analysis.

Regards, Paul

From: Rami Krispin @.> Sent: Thursday, October 24, 2024 9:43 PM To: RamiKrispin/ts-cluster-analysis-r @.> Cc: Prew, Paul @.>; Author @.> Subject: Re: [RamiKrispin/ts-cluster-analysis-r] Cannot follow the actions taken in the tutorial using RStudio (Issue #1)

All the data used in the workshop should be stored in the repository data folder. The error you are getting is related to the eia_metadata function from the EIAapi library. Here is how it is being used in the notebook: meta_gas_raw <- eia_metadata( ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd

All the data used in the workshop should be stored in the repository data folderhttps://urldefense.com/v3/__https:/github.com/RamiKrispin/ts-cluster-analysis-r/tree/main/data__;!!Nkc5UzxO!qjUb4QH4hxMLiLGR_7GjaAv2hpMBM9QZKbYHoWwUqS0cRnQVVd0SWmpbg8xAieHIBS8yqXOlyzx1_lo51qp5gkFLnA$. The error you are getting is related to the eia_metadata function from the EIAapi library. Here is how it is being used in the notebook:

meta_gas_raw <- eia_metadata(

api_path = "natural-gas/cons/sum/data/facet/parent",

api_key = api_key

)

Please make sure that you register to the API and you have a valid API key and assign it to the api_key variable.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/RamiKrispin/ts-cluster-analysis-r/issues/1*issuecomment-2436697284__;Iw!!Nkc5UzxO!qjUb4QH4hxMLiLGR_7GjaAv2hpMBM9QZKbYHoWwUqS0cRnQVVd0SWmpbg8xAieHIBS8yqXOlyzx1_lo51qrw7SAuAQ$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AB3IZYEBDKMQ5FD53HOB7Q3Z5GV3NAVCNFSM6AAAAABQPTUBU2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZWGY4TOMRYGQ__;!!Nkc5UzxO!qjUb4QH4hxMLiLGR_7GjaAv2hpMBM9QZKbYHoWwUqS0cRnQVVd0SWmpbg8xAieHIBS8yqXOlyzx1_lo51qoui_BSRQ$. You are receiving this because you authored the thread.Message ID: @.**@.>>

CONFIDENTIALITY NOTICE: This e-mail communication and any attachments may contain proprietary and privileged information for the use of the designated recipients named above. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

RamiKrispin commented 3 weeks ago

It seems that the library is either not install or not loaded, try to run the following before:

library(EIAapi)
Iowastater commented 3 weeks ago

Hello Rami, thank you for the suggestion. I spoke with a data scientist here. She said that she believes the problem is with Sys.getenv("LxL4jhAqs3LJvoZ0HMPd0l0OXb4g3jocDwcUCg0z")

install.packages("EIAapi") library(EIAapi)

api_key <- Sys.getenv("LxL4jhAqs3LJvoZ0HMPd0l0OXb4g3jocDwcUCg0z") api_key [1] "" meta_gas_raw <- eia_metadata(

  • api_path = "natural-gas/cons/sum/data/facet/parent",
  • api_key = api_key
  • ) Error in eia_metadata(api_path = "natural-gas/cons/sum/data/facet/parent", : Could not parse the metadata JSON... ❌ Please check the query parameters (e.g., api key, path, etc.) or the error log In addition: Warning message: In system(command = query, intern = TRUE) : running command 'curl 'https://api.eia.gov/v2/natural-gas/cons/sum/data/facet/parent'?api_key=' had status 3

Two questions –

  1. Do you have any advice for accessing the data using the API?

  2. Is it possible for you to just send the data “meta_gas_raw“ in a separate file?

    • I honestly don’t care about learning how to use the API or Docker or Quarto or whatever is needed besides R.
    • My only purpose in taking the tutorial was in learning how to do Time Series Clustering using R.

Thank you, Paul Prew

From: Rami Krispin @.> Sent: Friday, October 25, 2024 10:07 PM To: RamiKrispin/ts-cluster-analysis-r @.> Cc: Prew, Paul @.>; Author @.> Subject: Re: [RamiKrispin/ts-cluster-analysis-r] Cannot follow the actions taken in the tutorial using RStudio (Issue #1)

It seems that the library is either not install or not loaded, try to run the following before: library(EIAapi) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread. Message ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd

It seems that the library is either not install or not loaded, try to run the following before:

library(EIAapi)

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/RamiKrispin/ts-cluster-analysis-r/issues/1*issuecomment-2439204239__;Iw!!Nkc5UzxO!sanetu1XTCVUX7oJTYIj4WVAOFcUakahm3etrXRSX3v2d-dqtwBcI1138HSyeIxR4YR-Dr4c1E1Uin3JuG77rSeDTw$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AB3IZYDHO3B3FJY7A7UYBRLZ5MBMRAVCNFSM6AAAAABQPTUBU2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZZGIYDIMRTHE__;!!Nkc5UzxO!sanetu1XTCVUX7oJTYIj4WVAOFcUakahm3etrXRSX3v2d-dqtwBcI1138HSyeIxR4YR-Dr4c1E1Uin3JuG5NaGTisA$. You are receiving this because you authored the thread.Message ID: @.***>

CONFIDENTIALITY NOTICE: This e-mail communication and any attachments may contain proprietary and privileged information for the use of the designated recipients named above. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

Iowastater commented 3 weeks ago

I asked Microsoft Copilot – it appears that another command is necessary to set the environment

It looks like the environment variable for your EIA API key is not set. You’ll need to set this variable before you can use it in your script. Here’s how you can do it in R:

  1. Set the Environment Variable: R Sys.setenv(EIA_API_KEY = "your_actual_api_key") AI-generated code. Review and use carefully. More info on FAQhttps://www.bing.com/new#faq.

  2. Verify the Environment Variable: R Sys.getenv("EIA_API_KEY")

Sys.setenv(EIA_API_KEY = "LxL4jhAqs3LJvoZ0HMPd0l0OXb4g3jocDwcUCg0z")

api_key <- Sys.getenv("EIA_API_KEY") api_key [1] "LxL4jhAqs3LJvoZ0HMPd0l0OXb4g3jocDwcUCg0z" api_path_gas <- "natural-gas/cons/sum/data/"

meta_gas_raw <- eia_metadata(

  • api_path = "natural-gas/cons/sum/data/facet/parent",
  • api_key = api_key
  • ) Error in eia_metadata(api_path = "natural-gas/cons/sum/data/facet/parent", : Could not parse the metadata JSON... ❌ Please check the query parameters (e.g., api key, path, etc.) or the error log In addition: Warning message: In system(command = query, intern = TRUE) : running command 'curl 'https://api.eia.gov/v2/natural-gas/cons/sum/data/facet/parent'?api_key=LxL4jhAqs3LJvoZ0HMPd0l0OXb4g3jocDwcUCg0z' had status 3 meta_gas <- meta_gas_raw$data Error: object 'meta_gas_raw' not found

From: Prew, Paul Sent: Monday, October 28, 2024 3:39 PM To: RamiKrispin/ts-cluster-analysis-r @.>; RamiKrispin/ts-cluster-analysis-r @.> Cc: Author @.***> Subject: RE: [RamiKrispin/ts-cluster-analysis-r] Cannot follow the actions taken in the tutorial using RStudio (Issue #1)

Hello Rami, thank you for the suggestion. I spoke with a data scientist here. She said that she believes the problem is with Sys.getenv("LxL4jhAqs3LJvoZ0HMPd0l0OXb4g3jocDwcUCg0z")

install.packages("EIAapi") library(EIAapi)

api_key <- Sys.getenv("LxL4jhAqs3LJvoZ0HMPd0l0OXb4g3jocDwcUCg0z") api_key [1] "" meta_gas_raw <- eia_metadata(

  • api_path = "natural-gas/cons/sum/data/facet/parent",
  • api_key = api_key
  • ) Error in eia_metadata(api_path = "natural-gas/cons/sum/data/facet/parent", : Could not parse the metadata JSON... ❌ Please check the query parameters (e.g., api key, path, etc.) or the error log In addition: Warning message: In system(command = query, intern = TRUE) : running command 'curl 'https://api.eia.gov/v2/natural-gas/cons/sum/data/facet/parent'?api_key=' had status 3

Two questions –

  1. Do you have any advice for accessing the data using the API?

  2. Is it possible for you to just send the data “meta_gas_raw“ in a separate file?

    • I honestly don’t care about learning how to use the API or Docker or Quarto or whatever is needed besides R.
    • My only purpose in taking the tutorial was in learning how to do Time Series Clustering using R.

Thank you, Paul Prew

From: Rami Krispin @.> Sent: Friday, October 25, 2024 10:07 PM To: RamiKrispin/ts-cluster-analysis-r @.> Cc: Prew, Paul @.>; Author @.> Subject: Re: [RamiKrispin/ts-cluster-analysis-r] Cannot follow the actions taken in the tutorial using RStudio (Issue #1)

It seems that the library is either not install or not loaded, try to run the following before: library(EIAapi) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread. Message ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd

It seems that the library is either not install or not loaded, try to run the following before:

library(EIAapi)

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/RamiKrispin/ts-cluster-analysis-r/issues/1*issuecomment-2439204239__;Iw!!Nkc5UzxO!sanetu1XTCVUX7oJTYIj4WVAOFcUakahm3etrXRSX3v2d-dqtwBcI1138HSyeIxR4YR-Dr4c1E1Uin3JuG77rSeDTw$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AB3IZYDHO3B3FJY7A7UYBRLZ5MBMRAVCNFSM6AAAAABQPTUBU2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZZGIYDIMRTHE__;!!Nkc5UzxO!sanetu1XTCVUX7oJTYIj4WVAOFcUakahm3etrXRSX3v2d-dqtwBcI1138HSyeIxR4YR-Dr4c1E1Uin3JuG5NaGTisA$. You are receiving this because you authored the thread.Message ID: @.***>

CONFIDENTIALITY NOTICE: This e-mail communication and any attachments may contain proprietary and privileged information for the use of the designated recipients named above. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

RamiKrispin commented 3 weeks ago

The data is available in the repo under the data folder. Per my above comment:

Please make sure that you register to the API and you have a valid API key and assign it to the api_key variable.
Iowastater commented 3 weeks ago

Hello Rami, thank you for your reply. Please allow me to address your 2 comments. Both of your comments have been addressed previously, but I will reiterate the answers.

Comment #1: “The data is available in the repo under the data folder”

As I demonstrated in my initial email on 10/24, the first data file that is used in your code is called “eia_metadata”. You refer to the “repo” - which I am assuming is the zip repository of files sent as part of the tutorial.

The zip file has a data folder, but there is no file with the name eia_metadata. You created this “data” folder, so can you please provide more guidance on where exactly the eia_metadata is within it? I have been using R for 25 years. Is it possible you could provide the exact path using R code, because I don’t know how to follow a path that is just named “repo”.

@.***

Comment #2: “Please make sure that you register to the API and you have a valid API key and assign it to the api_key variable.”

How do I verify the key is valid? Below I will show how I have repeatedly tried to validate the key – LxL4jhAqs3LJvoZ0HMPd0l0OXb4g3jocDwcUCg0z Point A: I have gone to the EIA website repeatedly Opendata - U.S. Energy Information Administration (EIA)https://www.eia.gov/opendata/ – they keep sending me the same key

@.*** ==> Then I get the email below: “Your EIA API Key

The following is your unique EIA API key to use with the EIA API.

LxL4jhAqs3LJvoZ0HMPd0l0OXb4g3jocDwcUCg0z”

Point B: I gave you the key so you could try it yourself. Then you would know that it is being validated in a way that matches your definition of valid.

api_key <- Sys.getenv("EIA_API_KEY") api_key [1] "LxL4jhAqs3LJvoZ0HMPd0l0OXb4g3jocDwcUCg0z"

I have spent hours trying to get this R code to work, along with another data scientist. If my attempts do not make sense to you, could you please just attach the data file eia_metadata and send it to me directly. I can then load it into R and begin to learn how to do time series clustering using the R language.

Regards, Paul

From: Rami Krispin @.> Sent: Monday, October 28, 2024 9:35 PM To: RamiKrispin/ts-cluster-analysis-r @.> Cc: Prew, Paul @.>; Author @.> Subject: Re: [RamiKrispin/ts-cluster-analysis-r] Cannot follow the actions taken in the tutorial using RStudio (Issue #1)

The data is available in the repo under the data folder. Per my above comment: Please make sure that you register to the API and you have a valid API key and assign it to the api_key variable. — Reply to this email directly, view it on ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd

The data is available in the repo under the data folder. Per my above comment:

Please make sure that you register to the API and you have a valid API key and assign it to the api_key variable.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/RamiKrispin/ts-cluster-analysis-r/issues/1*issuecomment-2443043986__;Iw!!Nkc5UzxO!qwtDP-vNOYlihxLron-15iCYjioTgSIkU7Vu7PyAea2EzIcpO_gL80TVjnqX9vY12Ukn91OVzsKUzDrh0I8s_L-uVw$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AB3IZYBGZHCWZ6LGNIJRDBDZ53X5HAVCNFSM6AAAAABQPTUBU2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBTGA2DGOJYGY__;!!Nkc5UzxO!qwtDP-vNOYlihxLron-15iCYjioTgSIkU7Vu7PyAea2EzIcpO_gL80TVjnqX9vY12Ukn91OVzsKUzDrh0I9ZJPrRDg$. You are receiving this because you authored the thread.Message ID: @.***>

CONFIDENTIALITY NOTICE: This e-mail communication and any attachments may contain proprietary and privileged information for the use of the designated recipients named above. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.