TampaDevs / talent.tampa.dev

The reverse job board for Tampa Bay's Developer Community
https://talent.tampa.dev
Other
8 stars 2 forks source link

Unable to add Developer Profile #149

Closed dustin-lennon closed 3 months ago

dustin-lennon commented 3 months ago

Describe the bug

I am unable to add my developer profile due to the following errors being displayed:

image

To reproduce

Fill out the form as indicated. I used the following details: image

Click "Save" at the bottom of the page.

Expected behavior

New developer profile created to be used on the site.

Actual behavior

City can't be found (API error?) and it is looking for UTC and time zone which are not fields to be set.

Screenshots

Screenshots attached above.

chtzvt commented 3 months ago

@dustin-lennon Thanks for opening this issue, I'm looking into it.

Per my experimentation in a Codespace, it seems to be related to the way we're using the geocoder gem, which provides APIs that the application uses to validate and normalize locations.

The full output of my irb console session is included in the latter half of this issue (for clarity + reproducibility), but the main blocker in your case is this error during model validation while creating a new Developer:

Tests, by line number:

  1. Loads the seeds helper (tools to quickly create placeholder models for tests)
  2. Instantiates a Location model for Ocala, Florida
  3. missing_fields? shows us that the Location we created is valid.
  4. When trying to create a new Developer model instance with our test location, we can observe a series of Location-related validation errors in the output.
[25] pry(main)> require File.join(Rails.application.root, 'lib/seeds_helper.rb')
[26] pry(main)> location = Location.new(city: "Ocala", state: "Florida", country: "United States")
[27] pry(main)> location.missing_fields?
=> false
[28] pry(main)> dev = SeedsHelper.create_developer!("OcalaDev3", attributes: { location: location } )
  User Load (0.2ms)  SELECT "users".* FROM "users" WHERE "users"."email" = $1 LIMIT $2  [["email", "OcalaDev3@example.com"], ["LIMIT", 1]]
  TRANSACTION (0.2ms)  BEGIN
  User Exists? (0.3ms)  SELECT 1 AS one FROM "users" WHERE "users"."email" = $1 LIMIT $2  [["email", "ocaladev3@example.com"], ["LIMIT", 1]]
  User Create (0.3ms)  INSERT INTO "users" ("email", "encrypted_password", "reset_password_token", "reset_password_sent_at", "remember_created_at", "confirmation_token", "confirmed_at", "confirmation_sent_at", "unconfirmed_email", "created_at", "updated_at", "admin", "authentication_token", "suspended", "referral_code", "referrals_count") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16) RETURNING "id"  [["email", "ocaladev3@example.com"], ["encrypted_password", "[FILTERED]"], ["reset_password_token", "[FILTERED]"], ["reset_password_sent_at", "[FILTERED]"], ["remember_created_at", nil], ["confirmation_token", "[FILTERED]"], ["confirmed_at", "2024-05-28 00:58:37.257223"], ["confirmation_sent_at", nil], ["unconfirmed_email", nil], ["created_at", "2024-05-28 00:58:37.489790"], ["updated_at", "2024-05-28 00:58:37.489790"], ["admin", false], ["authentication_token", "[FILTERED]"], ["suspended", false], ["referral_code", nil], ["referrals_count", 0]]
  TRANSACTION (5.8ms)  COMMIT
  Developer Load (0.2ms)  SELECT "developers".* FROM "developers" WHERE "developers"."user_id" = $1 LIMIT $2  [["user_id", 38], ["LIMIT", 1]]
  TRANSACTION (0.1ms)  BEGIN
Geocoding API's response was not valid JSON
  TRANSACTION (0.2ms)  ROLLBACK
Geocoding API's response was not valid JSON
ActiveRecord::RecordInvalid: Validation failed: Avatar is not attached, Location time zone can't be blank, Location UTC offset can't be blank, Location city can't be found
from /home/vscode/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/validations.rb:80:in `raise_validation_error'

Let's focus on this error:

Geocoding API's response was not valid JSON

Zeroing in on this further in the Codespace's IRB console, I tried running a simple lookup for Tampa:

[30] pry(main)> Geocoder.search('Tampa, Florida')
Geocoding API's response was not valid JSON

People who have run into this issue before said that configuring the Geocoder gem's User-Agent header will permit these queries to be processed.

So I gave that a shot:

[32] pry(main)> Geocoder.configure(http_headers: { "User-Agent" => "Tampa Devs Talent (https://talent.tampa.dev)" })
=> {:timeout=>3,
 :lookup=>:nominatim,
 :ip_lookup=>:ipinfo_io,
 :language=>:en,
 :http_headers=>{"User-Agent"=>"Tampa Devs Talent (https://talent.tampa.dev)"},
 :use_https=>false,
 :http_proxy=>nil,
 :https_proxy=>nil,
 :api_key=>nil,
 :cache=>nil,
 :cache_prefix=>nil,
 :basic_auth=>{},
 :logger=>:kernel,
 :kernel_logger_level=>2,
 :always_raise=>[],
 :units=>:mi,
 :distances=>:linear,
 :cache_options=>{:prefix=>"geocoder:"}}

And sure enough, it worked:

[34] pry(main)> Geocoder.search('Ocala Florida')
=> [#<Geocoder::Result::Nominatim:0x00007bbd821b7950
  @cache_hit=nil,
  @data=
   {"place_id"=>292817117,
    "licence"=>"Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright",
    "osm_type"=>"relation",
    "osm_id"=>119060,
    "lat"=>"29.1871986",
    "lon"=>"-82.1400923",
    "class"=>"boundary",
    "type"=>"administrative",
    "place_rank"=>16,
    "importance"=>0.4826813637086117,
    "addresstype"=>"city",
    "name"=>"Ocala",
    "display_name"=>"Ocala, Marion County, Florida, United States",
    "address"=>{"city"=>"Ocala", "county"=>"Marion County", "state"=>"Florida", "ISO3166-2-lvl4"=>"US-FL", "country"=>"United States", "country_code"=>"us"},
    "boundingbox"=>["29.1209142", "29.2337310", "-82.2517933", "-82.0619973"]}>]

What's more, all location-related validation errors are eliminated when we try to create another developer in Ocala, FL again:

[37] pry(main)> dev = SeedsHelper.create_developer!("OcalaDev5", attributes: { location: location } )
  User Load (0.3ms)  SELECT "users".* FROM "users" WHERE "users"."email" = $1 LIMIT $2  [["email", "OcalaDev5@example.com"], ["LIMIT", 1]]
  TRANSACTION (0.2ms)  BEGIN
  User Exists? (0.3ms)  SELECT 1 AS one FROM "users" WHERE "users"."email" = $1 LIMIT $2  [["email", "ocaladev5@example.com"], ["LIMIT", 1]]
  User Create (0.3ms)  INSERT INTO "users" ("email", "encrypted_password", "reset_password_token", "reset_password_sent_at", "remember_created_at", "confirmation_token", "confirmed_at", "confirmation_sent_at", "unconfirmed_email", "created_at", "updated_at", "admin", "authentication_token", "suspended", "referral_code", "referrals_count") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16) RETURNING "id"  [["email", "ocaladev5@example.com"], ["encrypted_password", "[FILTERED]"], ["reset_password_token", "[FILTERED]"], ["reset_password_sent_at", "[FILTERED]"], ["remember_created_at", nil], ["confirmation_token", "[FILTERED]"], ["confirmed_at", "2024-05-28 01:10:25.895651"], ["confirmation_sent_at", nil], ["unconfirmed_email", nil], ["created_at", "2024-05-28 01:10:26.129205"], ["updated_at", "2024-05-28 01:10:26.129205"], ["admin", false], ["authentication_token", "[FILTERED]"], ["suspended", false], ["referral_code", nil], ["referrals_count", 0]]
  TRANSACTION (5.6ms)  COMMIT
  Developer Load (0.2ms)  SELECT "developers".* FROM "developers" WHERE "developers"."user_id" = $1 LIMIT $2  [["user_id", 39], ["LIMIT", 1]]
ActiveRecord::RecordInvalid: Validation failed: Avatar is not attached
from /home/vscode/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/validations.rb:80:in `raise_validation_error'

What we've learned is:

My recommended way to fix this would be to create an initializer (in config/initializers) with the following content:

Geocoder.configure(http_headers: { "User-Agent" => "Tampa Devs Talent (https://talent.tampa.dev)" })

This will make sure that the Geocoder gem is configured correctly for the application while Rails is starting it up.

Contributing a Fix

We give a special "Source Contributor" badge to folks that work on/maintain the site:

image

If you open a pull request, I'd be happy to merge it and give you a Source Contributor badge.

DylanSp commented 3 months ago

@chtzvt Would it be worth putting some kind of rate limiting in for how the app uses the Geocoder gem when it calls an external API? The default API is OpenStreetMap's nominatim API, which per the usage policy, has a rate limit of 1 request/second. Alternatively, it looks like Geocoder can be configured to use other APIs; it looks like Geoapify has a free plan that allows 5 requests/second.

chtzvt commented 3 months ago

@DylanSp I don't think we're running into rate limiting issues here, it's just a documented requirement of the nominatim API:

Nominatim Usage Policy (aka Geocoding Policy) Requirements Provide a valid HTTP Referer or User-Agent identifying the application (stock User-Agents as set by http libraries will not do).

(from the API documentation)

In other words, I would expect the proposed fix to work for our use case.

DylanSp commented 3 months ago

@chtzvt I don't think that this particular issue is caused by rate limiting*, but I figured it might be a useful change to make when working on this part of the code, in case it causes a problem in the future. Up to you.

* Though I'm curious why it only started happening now; as far as I can tell, the Nominatim API's had that requirement for the User-Agent header for a while.

chtzvt commented 3 months ago

@DylanSp That's fair (in the event that we were making a large volume of calls to the API on a regular basis), but I don't think these lookups are occurring at a rate faster than 1 request/sec.

If it were to become a problem, I'd do some refactoring (probably in the form of a dedicated service object w/ an inbuilt caching layer in front of the Geocoder gem), but we can cross that bridge if #152 doesn't resolve this issue.

chtzvt commented 3 months ago

@dustin-lennon The fix just went into production, and I can confirm that everything is working as expected :)

image