mhudecheck
commented
4 years ago What operating system & version of R are you using? 1,000 records shouldn't be enough to fill more than a couple hundred MBs of ram at a time (if that), so my best guess is that something else is going on.
Ekibe
Thank you for the package! It is perfect for what I am trying to do.
I am using your revgeo package on about 15,000 records.
When running the package by itself: city_per_rec <- revgeo(NRM$lon[i], NRM$lat[i], provider = 'photon', API = NULL, output = 'frame', item = 'city’)
I get through about 1000 records and it hangs up. I first thought it was photon throttling but later realized it was not. I now believe it is a RAM issue so I wrote a separate rscript:
Library
library(rstudioapi) library(revgeo)
Load iteration
load('iter.RData’)
City Reverse Geocoding
for (j in 1:a) { for (i in (b+1):(b+100)) { if (!is.na(NRM$TOWN[i]) && !is.na(NRM$LONGITUDE)) { city_per_rec <- revgeo(NRM$lon[i], NRM$lat[i], provider = 'photon', API = NULL, output = 'frame', item = 'city’) NRM$ChCity[i] <- toupper(as.character(city_per_rec$city)) rm(city_per_rec) } } b <- (b+100) gc() print('Taking a Break!’) Sys.sleep(30) gc() }
Update iteration
iter <- iter+1 save(iter, file = 'iter.RData’)
Save Workspace Image
save.image()
Run This Script Again
if(iter < a) { restartSession(command = 'source("City.R")') }
This should clear the workspace, release the RAM, and restart R to run the code on 100 records at a time however this still hangs up my computer after about 1000 records.
I got through the data by manually doing chunks of 500 and restarting my computer so I am thinking it is a RAM fragmentation issue (running on only 6GB).
Are you able to provide any advise?
Thank you, Ethan