SEO by making Food Oasis web-crawler-compatible

[fancyham]

Importance: 3/5.

Currently our site doesn’t show up on search engines even though we have very relevant/the best information. This is an issue with our React framework but we should figure it out if we want to help food-seekers.

It doesn’t have to be perfect SEO - just enough to show results for searches on organization names or addresses. Right now our listings are 100% invisible to Google.

This may be as simple as replacing our JavaScript buttons with explicit ahrefs.

Summary

FOLA barely appears on search engines. Why? Because our org listings are not visible to Google's web crawler.

Only about 6 of our pages appear in Google’s search results — the static ones like About Us, but none of the dynamic ones like our org listings.

The listing pages don’t show up because our React-based site uses client-side rendering and their crawlers won’t do that. However, Google has said there are ways to allow their web crawlers to index our content.

Let's get our 600+ listings to appear on Google. Details below.

As it is now

Our food-provider/pantry/meal program detail pages don't show up in Google search results meaning that our site is not being indexed by web crawlers from Google or Archive.org.

This is a problem for food-seekers, for our visibility in search engines, and for historical/archival purposes.

Our site seems to currently require human interaction to get to listings which means that our food-provider info pages aren't being crawled or indexed by search engines. This is probably partially due to the React-based user interface.

Goal

**To allow people to find listings using Google and other search engines, we should make our site web-crawler friendly. This might mean using ahrefs instead of JavaScript for links, or generating a 'flat' version of our site that bots can crawl and index that redirects to deep-links to listing detail pages on our site.

Use Cases

• Someone googles for “food pantries in west covina” — it’d be great to have a FOLA list of neighborhood pantries appear in Google’s search results.
• Someone searches for “My friends house food pantry address” — goal is that the FOLA listing for “My Friend’s House LA” appears as one of the search results.

Background

Archive.org uses web-crawlers to visit and make copies of entire sites for posterity, however it seems our site is not compatible with web-crawlers. This means that while our landing pages are available, food-provider detail pages are not archived. https://web.archive.org/web/20220000000000*/foodoasis.la

These archives and search engine indexes are important for two reasons

• SEO: So people can find direct links to our content based on searching for an org name, for example.
• Archival purposes: So that we or others can see old versions of our data.

There is a small risk that Google may scrape our info and pull our data into their maps engine, but since our goal is to connect people with food, that's a risk we can/should take.

Action Items

• [x] Consider: are we being bad internet citizens if we put listings and their info online? I think not, but let's think about it. : it should be fine. Our listings are gathered from public-facing websites.
• [x] Evaluate if our site is indexable by archive.org and google. Try searches using content from one of our food-provider-org pages. Another way to check: does our site work when javascript is disabled? : No, it’s not currently indexable.
• [ ] Come up with solutions that would make our site indexable, with deep links to content, so that when people search on an org name, that one of our detail pages is in the search results. It doesn't have to be pretty.
• [ ] Implement solutions

Resources

React vs Google search crawler suggests using ahrefs for everything (not JavaScript buttons) https://support.google.com/webmasters/thread/15430775/hi-is-seo-going-to-be-an-issue-with-a-website-made-on-react-website-has-multiple-pages?hl=en

[staceyrebekahscott]

@Nechir-89 In the future, if you would like to work on an issue that is in the New Approval column, please let one of the PMs know so we can assess and prioritize properly. I am approving this and moving to In Progress.

[Nechir-89]

@staceyrebekahscott thanks for letting me know it, seems you changed the way developers are assigned to an issue because we were usually talk to team leaders (John and Hannah) before stepping up with new issues.

[staceyrebekahscott]

@Nechir-89 No worries, and there may be some workflow conflicts going on here. I'll get that sorted out with the Leads so there is no confusion.

[entrotech]

discussed on 11/14 in dev meeting. Suggested resources: https://developers.google.com/search/docs/fundamentals/get-started-developers?hl=en https://developers.google.com/search/docs/fundamentals/seo-starter-guide?hl=en&visit_id=638040804367265683-3650024786&rd=1 https://www.mediavine.com/mediavine-university-growing-traffic-podcast-transcript/ https://answerthepublic.com/ https://www.wordstream.com/meta-keyword

[Nechir-89]

@entrotech @hanapotski @staceyrebekahscott @fancyham

I created this figma white board for the SEO research to make work easy for all of us.

Currently within the board mainly you will see:

• All the big steps for improving SEO (on the left side of the board).
• And how google indexed foodoasis.la (on the right side of the board).

the research is still in progress and I will add more and update to the figma board as I go,

[fancyham]

Hi everyone — I wanted to clarify that my intended main goal for this issue is to allow our org detail screens to be indexed by Google. I’m hoping that our new deep-link feature makes this possible.

We have a ton of individual org listings — but none appear in Google’s search results.

I’m less concerned about “SEO” (at the level of keywords), though that would be a nice plus.

BTW, if anyone thinks that we shouldn’t allow our site to be indexable by Google, please post here so we can discuss. I think it makes sense, but there are also risks as described above (for example, scrapers copying our data).

[Nechir-89]

Hey @fancyham just want to make sure I'm getting it right, from the image do you want the details panel on the left side to be indexed by google?

[fancyham]

@Nechir-89, Yup - that's the idea. Hopefully Google (or other search engines) can index text on the left for this org and access this org's listing via a deep-link URL.

[Nechir-89]

Thanks Bryan, I think part of the solution is to checkout if we have been indexed by google and the answer is YES, to prove that, right in this link you can see almost all pages that have been indexed by google for foodoasis.la

food oasis's index pages by google

the problem as you mentioned earlier we can not see any pantry, food provider, or even food program in any of these indexed pages by google, I'm just doing research to find solution for this problem

[fancyham]

I suspect we'd need to do something like 'server-side rendering' to flatten out the website for the Google Indexer? Sounds like Google's indexer doesn't work well with ReactJS, so this basically creates dummy deep-link pages that it can index.

https://stackoverflow.com/a/52039854/3366821

[fancyham]

Hi @Nechir-89, Just checking in to see if you've discovered anything that can help or run into roadblocks? (I just heard some Google Analytics results which reminded me about this and the free marketing that Google Search could give us, but currently we're missing out on)

[jelenaUX]

Wearing my UX hat: Based on Google Analytics data, most visitors now arrive by directly typing in foodoasis.la into their URL bar and then proceed to look for pantries. ("Visitors" = US-based people who visit FOLA website on their phone and do not click on any obvious FOLA staff/volunteer links.) The only informational page that visitors consult is "About Us" page, perhaps because it is prominently featured in Google search results for FOLA. Only a small number of visitors (10%) click on "About Us" and for the most part these visitors do not continue to look for pantry info. So the following changes may increase the numbers of visitors to the website and improve their experiences:

• Making it possible for Google to index specific food pantries on FOLA website;
• Improving SEO description of "About Us" page to help set visitors' expectations before they reach FOLA website;
• Adding a button "Find Food" into the main text on the "About Us" page.

[jelenaUX]

Here is a brief (3-page) cohort report with my interpretation. It looks like SEO would likely result in many more organic visitors, and these visitors have great conversion rates.

[Nechir-89]

Hi everyone,

For this issue I already discussed with @entrotech @hanapotski , and I believe this issue needs to be breaking down into smaller tasks and issues, Google already indexed us very well for about 15 pages and we have older versions of FOA in Archive.org

[fancyham]

Okay, just a reminder that this issue’s goal is to get the individual listings indexed not the static pages (which are already indexed fine).

Use case:

• Someone googles for “food pantries in west covina” — it’d be great to have a FOLA list of neighborhood pantries appear in Google’s search results.
• Someone searches for “My friends house food pantry address” — goal is that the FOLA listing for “My Friend’s House LA” appears as one of Google’s search results.

[fancyham]

Hi @Nechir-89 - would it make sense to un-assign yourself from this issue at this point so it moves to the correct kanban column?

[Nechir-89]

Okay, I'll unassign myself from it!

[fancyham]

Hi there - checking in on this older but important issue — what's the status on this currently?

[fancyham]

This may be as simple as replacing our JavaScript buttons with explicit ahrefs.