Error when using pullData

[IshitaGopal]

I am using the example provided in the vignette to get data from phoenix_rt:

k <- '...My API Key...' subset1 <- pullData(utd_api_key = k, table_name = "phoenix_rt", country = list('canada','China'), start = '20171101', end = '20171102', T)

But I get the below error: Error in readLines(curl::curl(url_submit), warn = FALSE) : HTTP error 404.

[jimmylevell]

I am also experiencing the same issue, where you able to solve it?

[KateHyoung]

Can you contact @MarcusMMS regarding this issue? This guy is currently working on the package.

[jimmylevell]

of course, could you provide an email to reach her? Or should I wait for his response in this thread?

[KateHyoung]

I believe she will reach you soon.

[MarcusMMS]

I am looking into this now. I will reply when I arrive at a solution. Thank you.

---

From: KateHyoungah @.***> Sent: Wednesday, November 3, 2021 9:17 AM To: KateHyoung/UTDEventData Cc: Sianan, Marcus; Mention Subject: Re: [KateHyoung/UTDEventData] Error when using pullData (#18)

I believe she will reach you soon.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FKateHyoung%2FUTDEventData%2Fissues%2F18%23issuecomment-959206018&data=04%7C01%7CMarcus.Sianan%40UTDallas.edu%7C792e57e1bd474df01f6008d99ed4a94c%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C637715458476526750%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=n%2FPDiE5BWFGaffb8GtIva9Axj6RzWhwzd2C8dPDL0MY%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAPY34WRW4GAQC5RDR3QTHALUKE77LANCNFSM42H2FPLQ&data=04%7C01%7CMarcus.Sianan%40UTDallas.edu%7C792e57e1bd474df01f6008d99ed4a94c%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C637715458476536714%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Pf2Vqu0MdZcq5%2BKyGfDLT7ZqPb6CZB%2B6mD1LgXrOI9Y%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7CMarcus.Sianan%40UTDallas.edu%7C792e57e1bd474df01f6008d99ed4a94c%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C637715458476536714%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=q1hLJRPyh1%2FlCdo3o5vhIvpA%2FXfr9%2FeaFCcod6CZRzc%3D&reserved=0 or Androidhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7CMarcus.Sianan%40UTDallas.edu%7C792e57e1bd474df01f6008d99ed4a94c%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C637715458476546667%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=0p%2Fu75g1hbS6m7Cdp9up1D0WiVYu%2BcZtuTTOYZvbT6E%3D&reserved=0.

[MarcusMMS]

Hello, I am working with our computer science person to resolve the issue. In the meantime, try using this code that I wrote to pull data in the same way that the example in the vignette was intended to do; however, please let me know if you have issues with the code after trying to run it. Thank you.

k <- "ENTER_YOUR_API_KEY_HERE"

country <- returnCountries(table_name = "PHOENIX_RT", country = list('canada','China'))

t <- returnTimes("PHOENIX_RT", '20171101','20171102')

q <- andList(c(country, t))

CanadaChina <- sendQuery(k, "PHOENIX_RT", q, citation = TRUE)

[jimmylevell]

Hello, thank you a lot for your quick update and provision of a workaround.

I have tried to implement your workaround but the query always runs out of memory, leading to the following error message:

Error. Consider to increase your memory limit of R. Use getQuerySize() to see the data size estimated.

When I check the memory limit on the system it is set to inf

> memory.limit()
[1] Inf
Warning message:
'memory.limit()' is Windows-specific

I have tried to address this limitations by using the following commands, but without any effect:

memory.limit(size=24000)
devtools::install_github("krlmlr/ulimit")
ulimit::memory_limit(24000)

Do you have a solution for this error?

Ps. the system has 72GB of RAM which is mostly not used.

Thank you a lot again and kind regards

[MarcusMMS]

Hello and thank you very much for the information. I have passed it on to our computer science person and will update you when I hear back.

[MarcusMMS]

Hello, I just received the following message from our computer science person: “That is related to R memory in their local machine. The requested data is huge and it can’t be handled with their local R memory limits. Ask them to increase local memory for R and try again.”

Please let me know if this helps or not. Thank you.

[jimmylevell]

Sorry for my late response.
I have tried the approach of your colleague, but was not successful loading the dataset. During these tests I gained some further insights regarding the error. It seems that in general the server eventdata.utdallas.edu responds with a time-out. For example when executing the following URL https://eventdata.utdallas.edu/api/data_key&query=%7B%22$and%22:[%7B%22country_code%22:%7B%22$in%22:[%22PLW%22,%22BMU%22,%22PSE%22,%22COG%22,%22MAF%22,%22IOT%22,%22CAF%22,%22GMB%22,%22SOM%22,%22HKG%22,%22URY%22,%22BEL%22,%22BES%22,%22MRT%22,%22BTN%22,%22CZE%22,%22ERI%22,%22SVN%22,%22LTU%22,%22CIV%22,%22PHL%22,%22GUM%22,%22MLT%22,%22TCA%22,%22ESP%22,%22TUN%22,%22GHA%22,%22BLR%22,%22SGP%22,%22JAM%22,%22BRB%22,%22ROU%22,%22SGS%22,%22ALA%22,%22GUF%22,%22MYS%22,%22TJK%22,%22VIR%22,%22NGA%22,%22ATG%22,%22MUS%22,%22LBY%22,%22TON%22,%22CRI%22,%22TKM%22,%22ZWE%22,%22MDV%22,%22NOR%22,%22BLZ%22,%22DNK%22,%22AGO%22,%22DMA%22,%22BOL%22,%22GLP%22,%22ATA%22,%22SSD%22,%22MWI%22,%22IRQ%22,%22WSM%22,%22KHM%22,%22IND%22,%22EGY%22,%22CMR%22,%22CHL%22,%22GTM%22,%22GEO%22,%22SRB%22,%22ARM%22,%22RUS%22,%22MAC%22,%22NAM%22,%22LVA%22,%22BGD%22,%22COD%22,%22FIN%22,%22KAZ%22,%22IRN%22,%22SYC%22,%22GRD%22,%22CUW%22,%22PAK%22,%22SXM%22,%22ALB%22,%22PAN%22,%22BRN%22,%22NIU%22,%22DZA%22,%22BDI%22,%22COL%22,%22JEY%22,%22PRK%22,%22ARE%22,%22AUS%22,%22PYF%22,%22FRA%22,%22CHN%22,%22ZAF%22,%22TTO%22,%22LKA%22,%22TLS%22]%7D%7D,%7B%22date8%22:%7B%22$gte%22:%2220171101%22,%22$lte%22:%2220211123%22%7D%7D]%7D&datasource=phoenix_rt

The following error message appears:

[MarcusMMS]

Thank you for your message and the information. I will look into it.

[MarcusMMS]

Hello, my colleague said that the URL formation is incorrect (API key is mentioned with wrong attribute). Please see the area he circled in the image, and let me know if that helps. Thank you.