search

grizzthedj / smart_proxy_ipam

Foreman Smart Proxy plugin for integration with External IPAM providers
GNU General Public License v3.0
18 stars 20 forks source link

Should installation of this plugin work on Foreman 3.3? #56

Closed dmgeurts closed 8 months ago

dmgeurts commented 2 years ago

Hi,

I see 2 years ago this plugin needed a fair bit of work to get working and some things were pending Foreman updates. What's the current status, or should I not bother as Foreman has moved along too far since?

dmgeurts commented 2 years ago

Managed to get the foreman-proxy setup to load and detect the service. But am getting the following when trying to use it when adding a subnet to Foreman:

ERF12-3215 [ProxyAPI::ProxyException]: Unable to obtain subnet 10.0.0.0/24 from External IPAM. ([RestClient::InternalServerError]: 500 Internal Server Error) for proxy https://foreman.domain.com:9090/ipam
dmgeurts commented 2 years ago

foreman-proxy log entries:

2022-08-12T02:12:35 566d16d6 [I] Started GET /ipam/subnet/10.0.0.0/24 group=
2022-08-12T02:12:35 566d16d6 [W] Error processing request '566d16d6-e163-4b64-b398-3ad48b7c7c80: <NoMethodError>: undefined method `zero?' for nil:NilClass
/usr/local/share/gems/gems/smart_proxy_ipam-0.1.4/lib/smart_proxy_ipam/netbox/netbox_client.rb:49:in `get_ipam_subnet_by_cidr'
/usr/local/share/gems/gems/smart_proxy_ipam-0.1.4/lib/smart_proxy_ipam/netbox/netbox_client.rb:29:in `get_ipam_subnet'
/usr/local/share/gems/gems/smart_proxy_ipam-0.1.4/lib/smart_proxy_ipam/ipam_api.rb:100:in `block in <class:Api>'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `each'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `block in call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in `synchronize'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in `service'
/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2022-08-12T02:12:35 566d16d6 [I] Finished GET /ipam/subnet/10.0.0.0/24 with 500 (28.65 ms)

I think the issue may well be that subnets are now called prefixes as I don't see the term subnets in the Netbox API documentation. https://docs.netbox.dev/en/stable/rest-api/overview/

dmgeurts commented 2 years ago

A query that used to be GET /ipam/subnet/10.0.0.0/24, I think should now be: GET /api/ipam/ip-addresses/?q=&parent=10.0.0.0/24

On a Netbox installation the API can be inspected as follows: https://netbox.domain.com/api/ipam/?format=api

GET /api/ipam/?format=api
HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

{
    "aggregates": "https://netbox.domain.com/api/ipam/aggregates/?format=api",
    "asns": "https://netbox.domain.com/api/ipam/asns/?format=api",
    "fhrp-group-assignments": "https://netbox.domain.com/api/ipam/fhrp-group-assignments/?format=api",
    "fhrp-groups": "https://netbox.domain.com/api/ipam/fhrp-groups/?format=api",
    "ip-addresses": "https://netbox.domain.com/api/ipam/ip-addresses/?format=api",
    "ip-ranges": "https://netbox.domain.com/api/ipam/ip-ranges/?format=api",
    "prefixes": "https://netbox.domain.com/api/ipam/prefixes/?format=api",
    "rirs": "https://netbox.domain.com/api/ipam/rirs/?format=api",
    "roles": "https://netbox.domain.com/api/ipam/roles/?format=api",
    "route-targets": "https://netbox.domain.com/api/ipam/route-targets/?format=api",
    "service-templates": "https://netbox.domain.com/api/ipam/service-templates/?format=api",
    "services": "https://netbox.domain.com/api/ipam/services/?format=api",
    "vlan-groups": "https://netbox.domain.com/api/ipam/vlan-groups/?format=api",
    "vlans": "https://netbox.domain.com/api/ipam/vlans/?format=api",
    "vrfs": "https://netbox.domain.com/api/ipam/vrfs/?format=api"
}
dmgeurts commented 2 years ago

Or, could it be that my config files aren't read?

user@foreman:/usr/local/share/gems/gems/smart_proxy_ipam-0.1.4/lib$ grep /subnet . -R
./smart_proxy_ipam/ipam_api.rb:    get '/subnet/:address/:prefix/next_ip' do
./smart_proxy_ipam/ipam_api.rb:    get '/subnet/:address/:prefix' do
./smart_proxy_ipam/ipam_api.rb:    get '/groups/:group/subnets' do
./smart_proxy_ipam/ipam_api.rb:    get '/subnet/:address/:prefix/:ip' do
./smart_proxy_ipam/ipam_api.rb:    post '/subnet/:address/:prefix/:ip' do
./smart_proxy_ipam/ipam_api.rb:    delete '/subnet/:address/:prefix/:ip' do
./smart_proxy_ipam/phpipam/phpipam_client.rb:      subnets = @api_resource.get("sections/#{group[:id]}/subnets/")

user@foreman:/usr/local/share/gems/gems/smart_proxy_ipam-0.1.4/lib$ grep /ip-addresses . -R
./smart_proxy_ipam/netbox/netbox_client.rb:      url = "ipam/ip-addresses/?#{URI.encode_www_form({ address: ip })}"
./smart_proxy_ipam/netbox/netbox_client.rb:      response = @api_resource.post('ipam/ip-addresses/', data.to_json)
./smart_proxy_ipam/netbox/netbox_client.rb:      response = @api_resource.get("ipam/ip-addresses/?#{params}")
./smart_proxy_ipam/netbox/netbox_client.rb:      response = @api_resource.delete("ipam/ip-addresses/#{address_id}/")
dmgeurts commented 2 years ago

I'm confused. The request sent to my IPAM is incompatible with Netbox, but the interpreter calls the netbox_client.rb code for the reply from Netbox. So what is going wrong?

See the log in the 2nd comment. netbox_client.rb errors are thrown, yet the last line shows that a call is made for phpipam (according to ipam_api.rb) instead of Netbox (netbox_client.rb)

grizzthedj commented 2 years ago

Thanks @dmgeurts - A few questions:

  1. What version of Netbox are you running?
  2. Can you post the contents for your config/externalipam.yml and config/externalipam_netbox.yml files?
  3. When you create the Netbox subnet in the Foreman UI, are you associating the subnet with the External IPAM enabled proxy?
dmgeurts commented 2 years ago
  • What version of Netbox are you running?

We were on v3.2.0, but I've just upgraded to v3.3.2.

  • Can you post the contents for your config/externalipam.yml and config/externalipam_netbox.yml files?
~$ locate externalipam.yml
/etc/foreman-proxy/settings.d/externalipam.yml
/usr/local/share/gems/gems/smart_proxy_ipam-0.1.4/settings.d/externalipam.yml
/usr/local/share/gems/gems/smart_proxy_ipam-0.1.4/settings.d/externalipam.yml.example

~$ sudo diff /etc/foreman-proxy/settings.d/externalipam.yml /usr/local/share/gems/gems/smart_proxy_ipam-0.1.4/settings.d/externalipam.yml

~$ sudo cat /etc/foreman-proxy/settings.d/externalipam.yml
---
:enabled: true
:use_provider: externalipam_netbox

And externalipam_netbox.yml:

~$ locate externalipam_netbox.yml
/etc/foreman-proxy/settings.d/externalipam_netbox.yml
/usr/local/share/gems/gems/smart_proxy_ipam-0.1.4/settings.d/externalipam_netbox.yml
/usr/local/share/gems/gems/smart_proxy_ipam-0.1.4/settings.d/externalipam_netbox.yml.example

~$ sudo diff /etc/foreman-proxy/settings.d/externalipam_netbox.yml /usr/local/share/gems/gems/smart_proxy_ipam-0.1.4/settings.d/externalipam_netbox.yml

~$ sudo cat /etc/foreman-proxy/settings.d/externalipam_netbox.yml
---
:url: 'https://netbox.domain.com/'
:token: '********'
  • When you create the Netbox subnet in the Foreman UI, are you associating the subnet with the External IPAM enabled proxy?

Yes indeed. I believe that without it the ipam proxy plugin code won't fire and will just create a local entry not tied to the subnet recorded in Netbox. We get syslog entries as posted when trying to associate the subnet to an external IPAM enabled proxy. The IPAM enabled proxy and the Foreman server are currently one and the same machine.

Thank you for looking into this.

grizzthedj commented 2 years ago

Thanks @dmgeurts - The configs look fine. Let me see if I can replicate the issue using Netbox 3.3.2.

dmgeurts commented 1 year ago

@grizzthedj Any luck replicating things?

dmgeurts commented 1 year ago

I'm now on Foreman 3.4.0 and Netbox 3.3.6. The API calls still fail for me with the same error.

Foreman is installed on CentOS Stream 8.

dmgeurts commented 1 year ago

Just verified the correct API call URL using curl from Foreman and it works, so I have the right API token and permissions set on Netbox.

How I installed the plugin:

sudo gem install smart_proxy_ipam

sudo cp /usr/local/share/gems/gems/smart_proxy_ipam-0.1.4/settings.d/externalipam_netbox.yml.example /etc/foreman-proxy/settings.d/externalipam_netbox.yml sudo vi /etc/foreman-proxy/settings.d/externalipam_netbox.yml 

---
:url: 'https://netbox.domain.com'
:token: ******

sudo cp /usr/local/share/gems/gems/smart_proxy_ipam-0.1.4/settings.d/externalipam.yml.example /etc/foreman-proxy/settings.d/externalipam.yml sudo vi /etc/foreman-proxy/settings.d/externalipam.yml 

---
:enabled: true
:use_provider: externalipam_netbox

sudo vi /usr/share/foreman-proxy/bundler.d/Gemfile.local.rb 

gem 'smart_proxy_ipam'

sudo systemctl restart foreman-proxy

Am I missing something obvious?

dmgeurts commented 1 year ago

One other thing I've noticed under Infrastructure >> Smart Proxies >> Services is a spinning wheel for External IPAM:

image
grizzthedj commented 1 year ago

Hi @dmgeurts - My apologies for the delay here. It has been difficult to find any time. I have some issues in my local env to work though before I can try to replicate, but the call from Foreman to the Proxy would be GET /ipam/subnet/10.0.0.0/24. The "real" call to Netbox, from the Proxy, would be: GET /api/ipam/ip-addresses/?q=&parent=10.0.0.0/24.

dmgeurts commented 1 year ago

@grizzthedj So how do I start troubleshooting this? I've run a tcpdump but after the initial hits when I didn't have the right filters set so didn't see the HTTP header defaults (URL etc). I'm not seeing much. These are the foreman-proxy logs when trying to add a subnet:

2022-11-29T14:49:37 9a4f8085 [I] Started GET /ipam/subnet/10.255.12.0/24 group=
2022-11-29T14:49:37 9a4f8085 [W] Error processing request '9a4f8085-7386-4d25-be6a-6ddbdb9101ff: <JSON::ParserError>: 783: unexpected token at ''
/usr/share/ruby/json/common.rb:156:in `parse'
/usr/share/ruby/json/common.rb:156:in `parse'
/usr/local/share/gems/gems/smart_proxy_ipam-0.1.4/lib/smart_proxy_ipam/netbox/netbox_client.rb:48:in `get_ipam_subnet_by_cidr'
/usr/local/share/gems/gems/smart_proxy_ipam-0.1.4/lib/smart_proxy_ipam/netbox/netbox_client.rb:29:in `get_ipam_subnet'
/usr/local/share/gems/gems/smart_proxy_ipam-0.1.4/lib/smart_proxy_ipam/ipam_api.rb:100:in `block in <class:Api>'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1686:in `call'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1686:in `block in compile!'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1023:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1042:in `route_eval'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1023:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1071:in `block in process_route'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1069:in `catch'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1069:in `process_route'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1021:in `block in route!'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1018:in `each'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1018:in `route!'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1140:in `block in dispatch!'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1112:in `block in invoke'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1112:in `catch'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1112:in `invoke'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1135:in `dispatch!'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:949:in `block in call!'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1112:in `block in invoke'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1112:in `catch'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1112:in `invoke'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:949:in `call!'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:938:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-2.2.2/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-2.2.2/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-2.2.2/lib/rack/protection/json_csrf.rb:26:in `call'
/usr/share/gems/gems/rack-protection-2.2.2/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.2.2/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.2.2/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-2.2.4/lib/rack/null_logger.rb:11:in `call'
/usr/share/gems/gems/rack-2.2.4/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/show_exceptions.rb:22:in `call'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:218:in `call'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1993:in `call'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1553:in `block in call'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1769:in `synchronize'
/usr/share/gems/gems/sinatra-2.2.2/lib/sinatra/base.rb:1553:in `call'
/usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/rack-2.2.4/lib/rack/builder.rb:244:in `call'
/usr/share/gems/gems/rack-2.2.4/lib/rack/handler/webrick.rb:95:in `service'
/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2022-11-29T14:49:37 9a4f8085 [I] Finished GET /ipam/subnet/10.255.12.0/24 with 500 (8.82 ms)

If I do a netcat to the Foreman proxy hostname on port 9090 it succeeds. Yet this is the error it throws in the GUI:

image
dmgeurts commented 1 year ago

I've also noticed the following in proxy.log when I try to do a curl to http(s)://foreman.domain.com:9090/ipam:

2022-11-29T14:22:54  [E] <OpenSSL::SSL::SSLError> SSL_accept returned=1 errno=0 state=error: http request
        /usr/share/ruby/webrick/server.rb:299:in `accept'
        /usr/share/ruby/webrick/server.rb:299:in `block (2 levels) in start_thread'
        /usr/share/ruby/webrick/utils.rb:263:in `timeout'
        /usr/share/ruby/webrick/server.rb:297:in `block in start_thread'
        /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2022-11-29T14:23:05 393359c8 [I] Started GET /ipam
2022-11-29T14:23:05 393359c8 [I] Finished GET /ipam with 404 (0.42 ms)
2022-11-29T14:23:20 0599f96e [I] Started GET /ipam
2022-11-29T14:23:20 0599f96e [I] Finished GET /ipam with 404 (0.33 ms)
dmgeurts commented 1 year ago

Doing a TCP dump I don't see any requests being made, so this looks like a local issue to the Smart Proxy.

lravelo commented 1 year ago

@dmgeurts any luck getting this plugin working? I followed the same installation steps that you laid out earlier in this thread. Though when I do a curl https://foreman.domain.com:9090/ipam it just says Requested url not found

Dragonpark commented 1 year ago

I was hoping to add a provider to this for Bluecat, but am seeing the same issue. I am getting Requested url not found as well. I know there is an active pull request to get this plugin merged into the main project. Is this still being planned?

dmgeurts commented 1 year ago

I've not had this plugin working successfully yet. In all honestly, I need to check things after updating Foreman and Katello. As the plugin is no longer listed under the foreman-proxy last time I checked.

dmgeurts commented 1 year ago

Hi @dmgeurts - My apologies for the delay here. It has been difficult to find any time. I have some issues in my local env to work though before I can try to replicate, but the call from Foreman to the Proxy would be GET /ipam/subnet/10.0.0.0/24. The "real" call to Netbox, from the Proxy, would be: GET /api/ipam/ip-addresses/?q=&parent=10.0.0.0/24.

Has this plugin been integrated into Foreman? I've seen references alluding to this, but I'm still unable to get this plugin working. On Foreman 3.5.2 I'm still seeing this:

image
dmgeurts commented 1 year ago

I'd be happy to troubleshoot the issue, if only I knew how...

dmgeurts commented 1 year ago

Smart-proxy logs show the successful initialisation of both elements:

image
dmgeurts commented 8 months ago

Right, I've finally worked out why the plugin didn't work for me. I'm embarrassed to admit the simplicity of it and how long it took me to work it out. But, maybe it will save some other poor soul time if I explain what happened.

The Netbox access logs showed:

user@netbox:~$ sudo tail /var/log/nginx/access.log
10.2.0.12 - - [12/Feb/2024:16:48:12 +0100] "GET //api/ipam/prefixes/?status=active&prefix=10.2.3.0%2F24 HTTP/1.1" 302 0 "-" "Ruby"
10.2.0.12 - - [12/Feb/2024:17:08:35 +0100] "GET //api/ipam/prefixes/?status=active&prefix=10.2.3.0%2F24 HTTP/1.1" 302 0 "-" "Ruby"
10.2.0.12 - - [12/Feb/2024:17:15:15 +0100] "GET //api/ipam/prefixes/?status=active&prefix=10.1.0.0%2F24 HTTP/1.1" 302 0 "-" "Ruby"

Foreman showed a 500 error in the UI and Netbox gives a 302 error. Checking the API key from terminal on the Foreman server works fine and the request is correct. However, I failed to notice the significance of the double slashes //api/...!

The reason I ended up with the double slashes is the following config in /etc/foreman-proxy/settings.d/externalipam_netbox.yml: 

---
# url is the hostname and path of the Netbox instance
:url: 'https://netbox.domain.com/'

Note the trailing slash. After removing it everything works perfectly. Meanwhile, I'm now on Foreman 3.9, so yes this plugin works fine on Foreman 3.9 and probably everything in between 3.3 and 3.9 too, if you don't make my mistake that is!