Closed dustin-lennon closed 3 months ago
@dustin-lennon Thanks for opening this issue, I'm looking into it.
Per my experimentation in a Codespace, it seems to be related to the way we're using the geocoder gem, which provides APIs that the application uses to validate and normalize locations.
The full output of my irb console session is included in the latter half of this issue (for clarity + reproducibility), but the main blocker in your case is this error during model validation while creating a new Developer:
Tests, by line number:
missing_fields?
shows us that the Location we created is valid.[25] pry(main)> require File.join(Rails.application.root, 'lib/seeds_helper.rb')
[26] pry(main)> location = Location.new(city: "Ocala", state: "Florida", country: "United States")
[27] pry(main)> location.missing_fields?
=> false
[28] pry(main)> dev = SeedsHelper.create_developer!("OcalaDev3", attributes: { location: location } )
User Load (0.2ms) SELECT "users".* FROM "users" WHERE "users"."email" = $1 LIMIT $2 [["email", "OcalaDev3@example.com"], ["LIMIT", 1]]
TRANSACTION (0.2ms) BEGIN
User Exists? (0.3ms) SELECT 1 AS one FROM "users" WHERE "users"."email" = $1 LIMIT $2 [["email", "ocaladev3@example.com"], ["LIMIT", 1]]
User Create (0.3ms) INSERT INTO "users" ("email", "encrypted_password", "reset_password_token", "reset_password_sent_at", "remember_created_at", "confirmation_token", "confirmed_at", "confirmation_sent_at", "unconfirmed_email", "created_at", "updated_at", "admin", "authentication_token", "suspended", "referral_code", "referrals_count") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16) RETURNING "id" [["email", "ocaladev3@example.com"], ["encrypted_password", "[FILTERED]"], ["reset_password_token", "[FILTERED]"], ["reset_password_sent_at", "[FILTERED]"], ["remember_created_at", nil], ["confirmation_token", "[FILTERED]"], ["confirmed_at", "2024-05-28 00:58:37.257223"], ["confirmation_sent_at", nil], ["unconfirmed_email", nil], ["created_at", "2024-05-28 00:58:37.489790"], ["updated_at", "2024-05-28 00:58:37.489790"], ["admin", false], ["authentication_token", "[FILTERED]"], ["suspended", false], ["referral_code", nil], ["referrals_count", 0]]
TRANSACTION (5.8ms) COMMIT
Developer Load (0.2ms) SELECT "developers".* FROM "developers" WHERE "developers"."user_id" = $1 LIMIT $2 [["user_id", 38], ["LIMIT", 1]]
TRANSACTION (0.1ms) BEGIN
Geocoding API's response was not valid JSON
TRANSACTION (0.2ms) ROLLBACK
Geocoding API's response was not valid JSON
ActiveRecord::RecordInvalid: Validation failed: Avatar is not attached, Location time zone can't be blank, Location UTC offset can't be blank, Location city can't be found
from /home/vscode/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/validations.rb:80:in `raise_validation_error'
Let's focus on this error:
Geocoding API's response was not valid JSON
Zeroing in on this further in the Codespace's IRB console, I tried running a simple lookup for Tampa:
[30] pry(main)> Geocoder.search('Tampa, Florida')
Geocoding API's response was not valid JSON
People who have run into this issue before said that configuring the Geocoder gem's User-Agent
header will permit these queries to be processed.
So I gave that a shot:
[32] pry(main)> Geocoder.configure(http_headers: { "User-Agent" => "Tampa Devs Talent (https://talent.tampa.dev)" })
=> {:timeout=>3,
:lookup=>:nominatim,
:ip_lookup=>:ipinfo_io,
:language=>:en,
:http_headers=>{"User-Agent"=>"Tampa Devs Talent (https://talent.tampa.dev)"},
:use_https=>false,
:http_proxy=>nil,
:https_proxy=>nil,
:api_key=>nil,
:cache=>nil,
:cache_prefix=>nil,
:basic_auth=>{},
:logger=>:kernel,
:kernel_logger_level=>2,
:always_raise=>[],
:units=>:mi,
:distances=>:linear,
:cache_options=>{:prefix=>"geocoder:"}}
And sure enough, it worked:
[34] pry(main)> Geocoder.search('Ocala Florida')
=> [#<Geocoder::Result::Nominatim:0x00007bbd821b7950
@cache_hit=nil,
@data=
{"place_id"=>292817117,
"licence"=>"Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright",
"osm_type"=>"relation",
"osm_id"=>119060,
"lat"=>"29.1871986",
"lon"=>"-82.1400923",
"class"=>"boundary",
"type"=>"administrative",
"place_rank"=>16,
"importance"=>0.4826813637086117,
"addresstype"=>"city",
"name"=>"Ocala",
"display_name"=>"Ocala, Marion County, Florida, United States",
"address"=>{"city"=>"Ocala", "county"=>"Marion County", "state"=>"Florida", "ISO3166-2-lvl4"=>"US-FL", "country"=>"United States", "country_code"=>"us"},
"boundingbox"=>["29.1209142", "29.2337310", "-82.2517933", "-82.0619973"]}>]
What's more, all location-related validation errors are eliminated when we try to create another developer in Ocala, FL again:
[37] pry(main)> dev = SeedsHelper.create_developer!("OcalaDev5", attributes: { location: location } )
User Load (0.3ms) SELECT "users".* FROM "users" WHERE "users"."email" = $1 LIMIT $2 [["email", "OcalaDev5@example.com"], ["LIMIT", 1]]
TRANSACTION (0.2ms) BEGIN
User Exists? (0.3ms) SELECT 1 AS one FROM "users" WHERE "users"."email" = $1 LIMIT $2 [["email", "ocaladev5@example.com"], ["LIMIT", 1]]
User Create (0.3ms) INSERT INTO "users" ("email", "encrypted_password", "reset_password_token", "reset_password_sent_at", "remember_created_at", "confirmation_token", "confirmed_at", "confirmation_sent_at", "unconfirmed_email", "created_at", "updated_at", "admin", "authentication_token", "suspended", "referral_code", "referrals_count") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16) RETURNING "id" [["email", "ocaladev5@example.com"], ["encrypted_password", "[FILTERED]"], ["reset_password_token", "[FILTERED]"], ["reset_password_sent_at", "[FILTERED]"], ["remember_created_at", nil], ["confirmation_token", "[FILTERED]"], ["confirmed_at", "2024-05-28 01:10:25.895651"], ["confirmation_sent_at", nil], ["unconfirmed_email", nil], ["created_at", "2024-05-28 01:10:26.129205"], ["updated_at", "2024-05-28 01:10:26.129205"], ["admin", false], ["authentication_token", "[FILTERED]"], ["suspended", false], ["referral_code", nil], ["referrals_count", 0]]
TRANSACTION (5.6ms) COMMIT
Developer Load (0.2ms) SELECT "developers".* FROM "developers" WHERE "developers"."user_id" = $1 LIMIT $2 [["user_id", 39], ["LIMIT", 1]]
ActiveRecord::RecordInvalid: Validation failed: Avatar is not attached
from /home/vscode/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/validations.rb:80:in `raise_validation_error'
What we've learned is:
User-Agent
header, User-Agent
header to be configured,My recommended way to fix this would be to create an initializer (in config/initializers
) with the following content:
Geocoder.configure(http_headers: { "User-Agent" => "Tampa Devs Talent (https://talent.tampa.dev)" })
This will make sure that the Geocoder gem is configured correctly for the application while Rails is starting it up.
We give a special "Source Contributor" badge to folks that work on/maintain the site:
If you open a pull request, I'd be happy to merge it and give you a Source Contributor badge.
@chtzvt Would it be worth putting some kind of rate limiting in for how the app uses the Geocoder gem when it calls an external API? The default API is OpenStreetMap's nominatim
API, which per the usage policy, has a rate limit of 1 request/second. Alternatively, it looks like Geocoder can be configured to use other APIs; it looks like Geoapify has a free plan that allows 5 requests/second.
@DylanSp I don't think we're running into rate limiting issues here, it's just a documented requirement of the nominatim API:
Nominatim Usage Policy (aka Geocoding Policy) Requirements Provide a valid HTTP Referer or User-Agent identifying the application (stock User-Agents as set by http libraries will not do).
(from the API documentation)
In other words, I would expect the proposed fix to work for our use case.
@chtzvt I don't think that this particular issue is caused by rate limiting*, but I figured it might be a useful change to make when working on this part of the code, in case it causes a problem in the future. Up to you.
* Though I'm curious why it only started happening now; as far as I can tell, the Nominatim API's had that requirement for the User-Agent header for a while.
@DylanSp That's fair (in the event that we were making a large volume of calls to the API on a regular basis), but I don't think these lookups are occurring at a rate faster than 1 request/sec.
If it were to become a problem, I'd do some refactoring (probably in the form of a dedicated service object w/ an inbuilt caching layer in front of the Geocoder gem), but we can cross that bridge if #152 doesn't resolve this issue.
@dustin-lennon The fix just went into production, and I can confirm that everything is working as expected :)
Describe the bug
I am unable to add my developer profile due to the following errors being displayed:
To reproduce
Fill out the form as indicated. I used the following details:![image](https://github.com/TampaDevs/talent.tampa.dev/assets/92093/e8846e35-2d2b-4d21-819f-d0e5b69b5138)
Click "Save" at the bottom of the page.
Expected behavior
New developer profile created to be used on the site.
Actual behavior
City can't be found (API error?) and it is looking for UTC and time zone which are not fields to be set.
Screenshots
Screenshots attached above.