search

grizzthedj / smart_proxy_ipam

Foreman Smart Proxy plugin for integration with External IPAM providers
GNU General Public License v3.0
18 stars 20 forks source link

https problem with smart proxy #50

Closed stephenbcollier closed 3 years ago

stephenbcollier commented 3 years ago

Hi I'm getting the following. Seems to be a code issue with https

The system works with a http phpipam but not a https phpipam.

Any suggestions appreciated - thanks

externalipam.yml

:enabled: true

:externalipam: :phpipam: :url: 'https://ipam.xxxxx.net/' :user: 'foreman' :password: 'xxxxxxxxxxxxxxxxxxxxxxxxx'

Error processing request '4a41afe4-4d22-45f7-9b74-92a28d554673: : 765: unexpected token at '<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

400 Bad Request

Bad Request

Your browser sent a request that this server could not understand.
Reason: You're speaking plain HTTP to an SSL-enabled server port.
Instead use the HTTPS scheme to access this URL, please.

' /opt/rh/rh-ruby25/root/usr/share/ruby/json/common.rb:156:in parse' /opt/rh/rh-ruby25/root/usr/share/ruby/json/common.rb:156:inparse' /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/smart_proxy_ipam-0.0.22/lib/smart_proxy_ipam/phpipam/phpipam_client.rb:343:in authenticate' /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/smart_proxy_ipam-0.0.22/lib/smart_proxy_ipam/phpipam/phpipam_client.rb:29:ininitialize' /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/smart_proxy_ipam-0.0.22/lib/smart_proxy_ipam/phpipam/phpipam_api.rb:112:in new' /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/smart_proxy_ipam-0.0.22/lib/smart_proxy_ipam/phpipam/phpipam_api.rb:112:inblock in ' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1635:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1635:inblock in compile!' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:992:in block (3 levels) in route!' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1011:inroute_eval' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:992:in block (2 levels) in route!' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1040:inblock in process_route' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1038:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1038:inprocess_route' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:990:in block in route!' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:989:ineach' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:989:in route!' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1097:inblock in dispatch!' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:in block in invoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:incatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:in invoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1094:indispatch!' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:924:in block in call!' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:inblock in invoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:ininvoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:924:in call!' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:913:incall' /usr/share/foreman-proxy/lib/proxy/log.rb:103:in call' /usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:incall' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.0.3/lib/rack/protection/xss_header.rb:18:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.0.3/lib/rack/protection/path_traversal.rb:16:incall' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.0.3/lib/rack/protection/json_csrf.rb:26:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.0.3/lib/rack/protection/base.rb:50:incall' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.0.3/lib/rack/protection/base.rb:50:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.0.3/lib/rack/protection/frame_options.rb:31:incall' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/null_logger.rb:11:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/head.rb:12:incall' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/show_exceptions.rb:22:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:194:incall' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1958:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1502:inblock in call' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1729:in synchronize' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.0.3/lib/sinatra/base.rb:1502:incall' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:74:in block in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:58:ineach' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:58:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/builder.rb:244:incall' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/handler/webrick.rb:95:in service' /opt/rh/rh-ruby25/root/usr/share/ruby/webrick/httpserver.rb:140:inservice' /opt/rh/rh-ruby25/root/usr/share/ruby/webrick/httpserver.rb:96:in run' /opt/rh/rh-ruby25/root/usr/share/ruby/webrick/server.rb:307:inblock in start_thread' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

stephenbcollier commented 3 years ago

below seems to fix it. Sorry no patch but very busy today

/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/smart_proxy_ipam-0.0.22/lib/smart_proxy_ipam/phpipam/phpipam_client.rb

307       Net::HTTP.start(uri.hostname, uri.port,
308         :use_ssl => uri.scheme == 'https') {|http|
309         http.request(request)

319       Net::HTTP.start(uri.hostname, uri.port,
320         :use_ssl => uri.scheme == 'https') {|http|
321         http.request(request)

331       Net::HTTP.start(uri.hostname, uri.port,
332         :use_ssl => uri.scheme == 'https') {|http|
333         http.request(request)

342       response = Net::HTTP.start(auth_uri.hostname, auth_uri.port,
343         :use_ssl => auth_uri.scheme == 'https') { |http|
344         http.request(request)
345       }
grizzthedj commented 3 years ago

@stephenbcollier Sorry for the delayed reply.

This has already been fixed in the latest version(0.1.0), which was published about a month ago. Please be aware that the latest plugin will only work with Foreman version 2.3 or greater(due to a sizable refactoring).

foremantdl2020 commented 3 years ago

@grizzthedj - I have ipam 0.1.0 running on on foreman 2.3; saw this error in ipam dashboard: ERF12-5974 [ProxyAPI::ProxyException]: Unable to obtain groups from External IPAM. ([TypeError]: no implicit conversion of String into Integer)

foreman proxy log looks good w/o error.

please advise...thanks

grizzthedj commented 3 years ago

@foremantdl2020 You need to upgrade smart_proxy_ipam to the latest version, which is 0.1.4. This should resolve your issue.

foremantdl2020 commented 3 years ago

I do have 0.1.4 foreman_ipam (0.1.0) smart_proxy_ipam (0.1.4)

foremantdl2020 commented 3 years ago

I don't see exernalipam in feature.all

irb(main):083:0> Feature.all => #<ActiveRecord::Relation [#<Feature id: 1, name: "Templates", created_at: "2021-03-05 18:58:29", updated_at: "2021-03-05 18:58:29">, #<Feature id: 2, name: "TFTP", created_at: "2021-03-05 18:58:53", updated_at: "2021-03-05 18:58:53">, #<Feature id: 3, name: "DNS", created_at: "2021-03-05 18:58:53", updated_at: "2021-03-05 18:58:53">, #<Feature id: 4, name: "DHCP", created_at: "2021-03-05 18:58:53", updated_at: "2021-03-05 18:58:53">, #<Feature id: 5, name: "Puppet", created_at: "2021-03-05 18:58:53", updated_at: "2021-03-05 18:58:53">, #<Feature id: 6, name: "Puppet CA", created_at: "2021-03-05 18:58:53", updated_at: "2021-03-05 18:58:53">, #<Feature id: 7, name: "BMC", created_at: "2021-03-05 18:58:53", updated_at: "2021-03-05 18:58:53">, #<Feature id: 8, name: "Realm", created_at: "2021-03-05 18:58:53", updated_at: "2021-03-05 18:58:53">, #<Feature id: 9, name: "Facts", created_at: "2021-03-05 18:58:53", updated_at: "2021-03-05 18:58:53">, #<Feature id: 10, name: "Logs", created_at: "2021-03-05 18:58:53", updated_at: "2021-03-05 18:58:53">, ...]> irb(main):084:0> irb(main):085:0> irb(main):086:0> irb(main):087:0> SmartProxyFeature.all => #<ActiveRecord::Relation [#<SmartProxyFeature smart_proxy_id: 1, feature_id: 6, id: 1, capabilities: [], settings: {"puppet_url"=>"https://foremandev.example.com:8140", "use_provider"=>["puppetca_hostname_whitelisting", "puppetca_http_api"]}>, #<SmartProxyFeature smart_proxy_id: 1, feature_id: 5, id: 2, capabilities: [], settings: {"puppet_url"=>"https://foremandev.example.com:8140", "use_provider"=>["puppet_proxy_puppet_api"]}>, #<SmartProxyFeature smart_proxy_id: 1, feature_id: 10, id: 3, capabilities: [], settings: {}>, #<SmartProxyFeature smart_proxy_id: 1, feature_id: 13, id: 4, capabilities: [], settings: {}>, #<SmartProxyFeature smart_proxy_id: 1, feature_id: 12, id: 5, capabilities: [], settings: {"use_provider"=>"externalipam_phpipam"}>]> irb(main):088:0> irb(main):089:0> irb(main):090:0> SmartProxy.all => #<ActiveRecord::Relation [#<SmartProxy id: 1, name: "foremandev.example.com", url: "https://foremandev.example.com:8443", created_at: "2021-03-05 19:01:26", updated_at: "2021-03-05 19:01:26", expired_logs: "0">]> irb(main):091:0> irb(main):092:0>

grizzthedj commented 3 years ago

Not having any luck reproducing this with Netbox or phpIPAM. Can you post the full stack trace, or provide some more details about your setup?

foremantdl2020 commented 3 years ago

I have basic foreman and foreman proxy running on same server, fresh install of foreman 2.3, then follow https://github.com/grizzthedj/smart_proxy_ipam to install external ipam plugin: I clone foreman and smart proxy 2.3 branch to /var/tmp/git, since my foreman is at 2.3.

After following that instruction, I notice I did not see "External ipam" in smart proxies freature, so I did the following: 1. echo "gem 'smart_proxy_ipam', :path => '/var/tmp/git/smart_proxy_ipam'" > /usr/share/foreman-proxy/bundler.d/Gemfile.local.rb sudo /etc/init.d/foreman-proxy restart

2. echo "gem 'foreman_ipam', :path => '/var/tmp/git/foreman_ipam'" > /usr/share/foreman/bundler.d/Gemfile.local.rb sudo foreman-rake db:migrate sudo foreman-rake db:seed sudo systemctl restart foreman

After the doing the above 2 instructions, I refresh smart proxies and see the "External ipam" feature and External IPAM dashboard in infrastructure.

note: after running sudo foreman-rake db:seed,I notice one error display, below User with login admin already exists, not seeding as admin

here are proxy log and foreman log when I click on Infrastructure --> External IPAM

/var/log/foreman-proxy/proxy.log: 2021-03-09T20:43:11 d89b387e [I] Started GET /ipam/groups 2021-03-09T20:43:11 d89b387e [I] Starting ip cache maintenance for IPAM provider , used by /next_ip. 2021-03-09T20:43:11 d89b387e [I] Finished GET /ipam/groups with 200 (71.36 ms)

/var/log/foreman/production.log: 2021-03-09T20:43:11 [I|app|d89b387e] Started GET "/ipam" for 127.0.0.1 at 2021-03-09 20:43:11 +0000 2021-03-09T20:43:11 [I|app|d89b387e] Processing by ForemanIpam::ExternalIpamController#dashboard as HTML 2021-03-09T20:43:11 [I|app|d89b387e] Rendering /var/tmp/git/foreman_ipam/app/views/foreman_ipam/external_ipam/dashboard.html.erb within layouts/application 2021-03-09T20:43:11 [I|app|d89b387e] Rendered /var/tmp/git/foreman_ipam/app/views/foreman_ipam/external_ipam/dashboard.html.erb within layouts/application (Duration: 1.3ms | Allocations: 881) 2021-03-09T20:43:11 [I|app|d89b387e] Rendered layouts/_application_content.html.erb (Duration: 0.8ms | Allocations: 437) 2021-03-09T20:43:11 [I|app|d89b387e] Rendering layouts/base.html.erb 2021-03-09T20:43:11 [I|app|d89b387e] Rendered layouts/base.html.erb (Duration: 14.2ms | Allocations: 14068) 2021-03-09T20:43:11 [I|app|d89b387e] Completed 200 OK in 124ms (Views: 19.2ms | ActiveRecord: 2.9ms | Allocations: 23640) 2021-03-09T20:43:12 [I|app|3799111e] Started GET "/notification_recipients" for 127.0.0.1 at 2021-03-09 20:43:12 +0000 2021-03-09T20:43:12 [I|app|3799111e] Processing by NotificationRecipientsController#index as JSON 2021-03-09T20:43:12 [I|app|3799111e] Completed 200 OK in 6ms (Views: 0.1ms | ActiveRecord: 0.6ms | Allocations: 2273) 2021-03-09T20:43:22 [I|app|23415708] Started GET "/notification_recipients" for 127.0.0.1 at 2021-03-09 20:43:22 +0000 2021-03-09T20:43:22 [I|app|23415708] Processing by NotificationRecipientsController#index as JSON 2021-03-09T20:43:22 [I|app|23415708] Completed 200 OK in 5ms (Views: 0.1ms | ActiveRecord: 0.6ms | Allocations: 2125)

grizzthedj commented 3 years ago

A few questions:

  1. What External IPAM provider do you have configured? phpIPAM or Netbox?
  2. When Navigating to Infrastructure => IPAM Dashboard, do you see a list of groups? (i.e. phpIPAM sections or Netbox VRF's), or is there an error?
  3. Can you post the contents of your External IPAM configs? If you upgraded the plugin from an older version, the old configs won't work anymore. This may not be the case, but here is what your configs should look like, depending on the provider you are using:

externalipam.yml

---
:enabled: true
:use_provider: externalipam_phpipam
# :use_provider: externalipam_netbox

externalipam_phpipam.yml

---
:url: 'http://foreman-url'
:user: 'foreman'
:password: 'foreman'

externalipam_netbox.yml

---
:url: 'http://netbox-url'
:token: '9834h93g7938wgf749gf972fg92'
foremantdl2020 commented 3 years ago

I think your externalipam_phpipam.yml url is supposed to be ipam url, not foreman-url.

I use phpIPAM provider.

here is my ipam config

/etc/foreman-proxy/settings.d/externalipam.yml

:enabled: true :use_provider: externalipam_phpipam

/etc/foreman-proxy/settings.d/externalipam_phpipam.yml:

:url: https://ipam-test.example.com :user: test1 :password: password1

foremantdl2020 commented 3 years ago

When Navigating to Infrastructure => IPAM Dashboard, I see this error:

ERF12-5974 [ProxyAPI::ProxyException]: Unable to obtain groups from External IPAM. ([TypeError]: no implicit conversion of String into Integer) for proxy https://foremandev.example.com:8443/ipam

grizzthedj commented 3 years ago

Do you also have an API key created in phpIPAM? Your username(in externalipam_phpipam.yml) and the API Key in phpIPAM name must be exactly the same in order for the authentication to work.

foremantdl2020 commented 3 years ago

yes. I create user and api key (App Id: test1); both user and appi id are same: test1

foremantdl2020 commented 3 years ago

proxy successfully connect to ipam, from log: Finished GET /ipam/groups with 200 (66.51 ms)

if I change test1 password to be incorrect, then I get: Finished GET /ipam/groups with 500 (47.32 ms)

This mean foreman proxy connect to ipam ok.

and from ipam apache2 access log, it show successful commect from foreman: access log: [11/Mar/2021:20:27:21 +0000] "GET /api/test1/sections/ HTTP/1.1" 200 3621 "-" "Ruby"

foremantdl2020 commented 3 years ago

btw - my phpipam version is v1.4

foremantdl2020 commented 3 years ago

on apache2 (phpipam) access log, seem like get section return 200, ok.

[11/Mar/2021:21:14:26 +0000] "GET /api/test1/sections/ HTTP/1.1" 200 3909 "-" "Ruby"

grizzthedj commented 3 years ago

@foremantdl2020 Could you post the contents of your /path/to/foreman/lib/proxy_api/external_ipam.rb file?