Lab 03 - Visualizing Shapefiles GDAL Error 1 Issue

[nickmcmullen]

Hi Professor Stowers,

I ran into another issue when downloading shapefiles. When running the script and grabbing state & county level shapefiles, I receive the warning depicted below.

This does not stop the shapefile from downloading, as I am able to see the features when calling the object:

However, when trying to visualize the outline, I get the same recursive GDAL warning saying "cannot find proj.db"

I tried restarting R, clearing the environment, detaching and reloading tigris, and still ran into the same error. A search through stack exchange and other forums didn't yield much in the way of help either.

Appreciate you taking a look!

cc: @castower

[castower]

Hello @nickmcmullen,

Thanks for bringing this to my attention! Another student ran into the same error.

There have been recent updates to the sp, sf, and tigris packages from January 2024 to as recently as a few days ago for the sf package which may be the source of the problem. Unfortunately I cannot replicate it on my end, even with a clean install.

However, you can try installing older versions of the package(s) to see if that resolves the issue:

1) The tigris package imports the sf package which works with GDAL and could be causing issues. It was recently updated a few days ago: (https://cran.r-project.org/web/packages/sf/index.html). You can try to update it or install an older version.

To install an older version, you need to specify the URL to the archived version (the link below is the December 2023 one):

packageurl <- "https://cran.r-project.org/src/contrib/Archive/sf/sf_1.0-15.tar.gz"
utils::install.packages(packageurl, repos=NULL, type="source")

You can view all of the archived versions here (just right-click and select copy address to get the URL if you want to try a different version):
https://cran.r-project.org/src/contrib/Archive/sf/

2) Tigris also imports the sp package which was updated in January 2024 (https://cran.r-project.org/web/packages/sp/index.html). Similarly, you can try installing an older version of the package:

packageurl <- "https://cran.r-project.org/src/contrib/Archive/sp/sp_2.1-2.tar.gz"
utils::install.packages(packageurl, repos=NULL, type="source")

You can view all of the archived versions here:
https://cran.r-project.org/src/contrib/Archive/sp/

3) Finally, the most recent version of tigris was released in January 2024. Similar to the sf and sp packages, you can specify an older version of the package like the one from September and see if that resolves the issue:

packageurl <- "https://cran.r-project.org/src/contrib/Archive/tigris/tigris_2.0.4.tar.gz"
utils::install.packages(packageurl, repos=NULL, type="source")

You can view all of the archived versions here:
https://cran.r-project.org/src/contrib/Archive/tigris/

You can also review all of the associated packages for tigris here:
https://cran.r-project.org/web/packages/tigris/index.html

If not, I will provide code you can try with the tidycensus package as well as download and upload the necessary shapefiles here to the repo tomorrow.

[nickmcmullen]

@castower Unfortunately this did not work. I believe you're right that the issue sits with the SF package, but I am unable to even install earlier versions. I was able to install older versions of tigris and sp. Below is what I see when trying to install SF 1.0-15.

I also tried installing GDAL directly in the terminal via homebrew, via info from r-spatial/sf repo on Github. The install of GDAL and proj were successful, but the package still is not recognizing it. Very odd

[castower]

Hello @nickmcmullen, When you run the following do you get a file path returned?

sf::sf_proj_search_paths()

[nickmcmullen]

@castower Yes, here is what I see

[castower]

@nickmcmullen, Okay, I think that's the issue, it's pulling the proj path from outside of renv.

See if the following code works:

sf::sf_proj_search_paths(paths = here::here("renv/library/R-4.3/aarch64-apple-darwin20/sf/proj"))

[nickmcmullen]

@castower here is what I see when I run that

[castower]

@nickmcmullen

Okay, if you re-run the empty function, do you now see that path added to the list/replacing the others?

sf::sf_proj_search_paths()

[nickmcmullen]

@castower I sure do!

[castower]

@nickmcmullen okay, great! Now you should be able to pull the shapefiles/map them, but let me know if you're still having issues and we can check if we also need to set the GDAL path.

I think that should resolve the issue since it couldn't find proj.db and that file is stored in that folder (you can navigate to renv > library > R-4.3 > aarch64-apple-darwin20 > sf > proj under 'Files' in RStudio to see it).

You can also view the documentation here: https://r-spatial.github.io/sf/reference/proj_tools.html

[nickmcmullen]

@castower still running into the same problem.

[nickmcmullen]

Also navigating to the directory I don't see the proj file

[castower]

@nickmcmullen, Ah, strange! Okay, try removing and re-installing it and see if the proj folder appears. If not, remove it again and try re-installing it from source (this does take longer than the usual installation):

# Regular install
utils::remove.packages("sf")
utils::install.packages("sf")

# Install from source
utils::remove.packages("sf")
utils::install.packages( c("sf"), type="source")

[nickmcmullen]

@castower thanks. I can do the regular install but the proj folder still doesn't appear. When I try to install from source, I get this error

[castower]

@nickmcmullen,

Okay, try the following and replace ENTER_PATH with one of the PROJ paths that sf originally listed: https://github.com/Watts-College/paf-515-spr-2024/issues/7#issuecomment-2028087340

Let's see if that will pull it in with the install --- Mac is a little trickier for source installations than Windows which uses RTools. install.packages("sf", configure.args = "--with-proj-lib= ENTER_PATH ")

[nickmcmullen]

@castower so weird. I was able to run your line of code, but the error still persisted. Sorry about this! I didn't think this issue would be such a sticky one.

It looks like I am able to download the shape file, which is what I experienced before, but when visualizing I get this recursive error.

Maybe it would be best to see if I can schedule some office hours with you next week to try to tackle live?

[castower]

@nickmcmullen

Can you see the proj folder now in sf?

Upon further research, this seems to be a Mac-specific error conflicting with renv that occurs sometimes and sometimes does not without a clear cause (which is likely why I can't replicate it on Windows).

I'll do a bit more reading to see if there's any additional guidance I can find, but it may be necessary to just complete this particular lab outside of the RStudio project/renv environment.

To do that, you can close the project in the top right corner, navigate to the folder where you've stored your repo and use setwd() to specify your location as the main folder of the repo. The here() package will work just the same (it will start from the setwd location instead of searching for the .rproj file), git will still recognize the changes to your files, etc. but your code will pull from the R installation on your local machine instead of the renv one.

Looking at your previous code, your setwd() should look something like this:

setwd("/Users/nickmcmullin/Documents/RStudio Desktop/PAF515")

[nickmcmullen]

@castower thank you for researching for me. I still don't see the proj folder in the renv directory

Also, when I run getwd() I already see, without setting the working directory to a different destination:

Should my working directory be something else?

[castower]

@nickmcmullen, Nope, that's perfect! Do you have your RStudio Project open?

You'll want to see "None" in the top right corner to avoid working within renv:

[nickmcmullen]

UPDATE:: Disregard, I had to set the wd before loading the here package. I will follow up after I run through everything again. Thanks.

@castower When I pull out of the project here() fails because the directory is not correct. Even if I set the working directory before doing import::here()

Looks like the root directory for here() outside of the project is just Documents - but I can't figure out how to change it.

I tried changing it and here is the error I got...

[nickmcmullen]

@castower it worked! Amazing the hoops we had to jump through to get this to work on a Mac. Hopefully our other Mac users don't run into the same roadblocks.

Thank you so much for the help troubleshooting.

[castower]

@nickmcmullen Yay!!! Glad that worked!

I'm not sure what it is specifically about the mapping packages not playing nicely with renv on Mac, but I'll leave this issue open for reference in case anyone else has the same problem.

[nickmcmullen]

Awesome - thank you again!

[acampbell7]

I want to say thank you both for bringing this up and helping find a solution. This error occurred with me, definitely a long workaround hahah

[nickmcmullen]

Aw yay @acampbell7 so glad it helped.

[mkfiles1]

@castower I am not sure I completely understand the work around. I tried creating the lab outside of the project. I changed the setup fucntion to have the root directory of the project folder

[castower]

@mkfiles1, You shouldn't need to make any changes to knitr() (that only affects the knitting steps).

Instead, you will want to use setwd() to identify the location of your repo. Looking at the screenshot, it appears that should be something like:

setwd("/Users/user/Documents/PAF515/project-paf-515-2024s-team-01")

[mkfiles1]

I did initially try that but then I received the same error

[mkfiles1]

@castower I am not sure if this is still part of the same issue I think it is. I got my file pathway to work and it is creating the map now but I cannot get gg_int10 to run

[nickmcmullen]

@mkfiles1 to set your working directory try using detach() to remove the here package, then run the setwd() command before reattaching the package using library("here"). Then try getwd() again to see if the working directory is set you expected. You can also check by running here() after you have set the working directory and reattached the package.

[acampbell7]

@mkfiles1 for me I did : Update your .from = here::here in functions is udated. Anytime you pull for the working directory youll need to put , looks like in your case, "/Users/user/Documents/PAF515/project-paf-515-2024s-team-01/analysis/project_data_steps_moira.R" - and this will be for all the setwd other files. However, you will then need to updated your project steps file

Load US Census region data

census_regions <- readxl::read_excel(here::here("Users/user/Documents/PAF515/project-paf-515-2024s-team-01data/raw/Census_Data_SVI/census_regions.xlsx"))

Thats how it worked for me, hopefully that helps for you

[castower]

@nickmcmullen @acampbell7, Thank you so much for adding these helpful tips!

@mkfiles1,

Hopefully these suggestions will help!

Another thing to check:

Was your merge of the shapefile with the county_svi_flags10 data set successful? It should have a column called geometry.

[mkfiles1]

@castower @nickmcmullen @acampbell7 Thank you all so much for your help. I was able to get everything to work

[castower]

Yay!! @mkfiles1

[acampbell7]

@castower Good Evening - sorry for the late message- while uploading the shapefile to the website I received an error/only flags pop 2010/2020 give me an error. Is this due to the Macbook?

Thank you,

[castower]

Hello @acampbell7,

I was able to review your markdown file. You'll want to update the URL to match your team's website, it currently has the example website URL. Once you update that your maps should appear, but let me know if you run into any issues!

The website building process is all on GitHub so should work the same regardless of operating system.

[acampbell7]

Works thank you! Must have mis copy and pasted

[castower]

Great @acampbell7!