search

gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
17.59k stars 1.76k forks source link

Http Fortigate gui application access giving "Internal Server Error" #17707

Closed raphaelbkt closed 1 year ago

raphaelbkt commented 2 years ago

Hi,

I use Teleport's application access to join firewalls that are not directly accessible. I have configured multiple http and https application accesses that are working well.

However when I try the same to access Fortinet's gui (FortiOS) I have errors in logs and html page gives "Internal Server Error". I can ping the management interface of the firewall so routing id OK. I tried multiple combinations including whanging application uri and putting IP to join the service (I know the certificate is valid for the IP), putting "insecure_skip_verify", giving a public_addr for the entry but I still face tha same error.

Teleport version 10.3.1-1 on a Rocky Linux 8.6 Green Obsidian.

Here is the application config : 

- name: "vpn02"
  description: vpn02
  uri: "https://11.22.33.44:443"
  insecure_skip_verify
  public_addr "vpn02.xxx.admin"

If I make a SSH local port redirection I know I can use the Firewall's gui to configure it. Another info : I tried with mutliple Fortinets with multiple versions and sometimes I have aa timeout on the page, sometimes I manage to get first login page and then I have the internal error.

Here are the DEBUG logs I see:

2022-10-24T11:41:08+02:00 DEBU [AUTH]      ClientCertPool -> cert(app.xxx.admin issued by app.xxx.admin:264072570739875274449628954669027062524) auth/middleware.go:673
2022-10-24T11:41:08+02:00 DEBU [AUTH]      ClientCertPool -> cert(app.xxx.admin issued by app.xxx.admin:178967818217414070879805098676346956657) auth/middleware.go:673
2022-10-24T11:41:08+02:00 DEBU [WEB]       Creating application web session for fwcp1-d1-fw-client01.app.xxx.admin in app.xxx.admin. web/apps.go:166
2022-10-24T11:41:08+02:00 DEBU [KEYGEN]    generated user key for [admin -teleport-internal-join] with expiry on (1666647638) 2022-10-24 21:40:38.000256776 +0000 UTC native/native.go:258
2022-10-24T11:41:08+02:00 DEBU [AUTH]      Failed setting default kubernetes cluster for user login (user did not provide a cluster); leaving KubernetesCluster extension in the TLS certificate empty auth/auth.go:1363
2022-10-24T11:41:08+02:00 INFO [CA]        Generating TLS certificate {0x9a64708 0xc0022befc0 1.3.9999.1.7=#130d6170702e73626d2e61646d696e,1.3.9999.1.5=#130d6170702e73626d2e61646d696e,1.3.9999.1.6=#132266776370312d64312d66772d636c69656e7430312e6170702e73626d2e61646d696e,1.3.9999.1.4=#132435343066393337352d326336632d343037302d616138612d373231663763323033383464,CN=admin,OU=usage:apps,O=access,POSTALCODE={\"aws_role_arns\":null\,\"db_names\":null\,\"db_users\":null\,\"kubernetes_groups\":null\,\"kubernetes_users\":null\,\"logins\":[\"admin\"]\,\"windows_logins\":null},STREET=app.xxx.admin,L=admin+L=-teleport-internal-join 2022-10-24 21:40:38.002016877 +0000 UTC [] [] 5 []}. common_name:admin dns_names:[] locality:[admin -teleport-internal-join] not_after:2022-10-24 21:40:38.002016877 +0000 UTC org:[access] org_unit:[usage:apps] tlsca/ca.go:874
2022-10-24T11:41:08+02:00 INFO [AUDIT]     cert.create cert_type:user cluster_name:app.xxx.admin code:TC000I ei:0 event:cert.create identity:map[expires:2022-10-24T21:40:38.002016877Z logins:[admin -teleport-internal-join] roles:[access] route_to_app:map[cluster_name:app.xxx.admin name: public_addr:fwcp1-d1-fw-client01.app.xxx.admin session_id:540f9375-2c6c-4070-aa8a-721f7c20384d] route_to_cluster:app.xxx.admin teleport_cluster:app.xxx.admin traits:map[aws_role_arns:<nil> db_names:<nil> db_users:<nil> kubernetes_groups:<nil> kubernetes_users:<nil> logins:[admin] windows_logins:<nil>] usage:[usage:apps] user:admin] time:2022-10-24T09:41:08.891Z uid:7ad921f9-1936-4fff-b3f2-7c3922beba57 events/emitter.go:263
2022-10-24T11:41:08+02:00 DEBU [AUTH]      Generated application web session for admin with TTL 11h59m29.113100978s. auth/sessions.go:106
2022-10-24T11:41:09+02:00 INFO [AUDIT]     app.session.start addr.remote:192.168.221.201:40388 app_name:fwcp1-d1-fw-client01 app_public_addr:fwcp1-d1-fw-client01.app.xxx.admin app_uri:https://10.125.14.31/ng cluster_name:app.xxx.admin code:T2007I ei:0 event:app.session.start namespace:default public_addr:fwcp1-d1-fw-client01.app.xxx.admin server_id:bf4a4dd1-21f6-4c85-a43f-720f529b76b6 sid:540f9375-2c6c-4070-aa8a-721f7c20384d time:2022-10-24T09:41:09.619Z uid:e961b5f0-3baa-4faf-8da7-9e09b9d68786 user:admin events/emitter.go:263
2022-10-24T11:41:09+02:00 INFO [AUDIT]     app.session.start addr.remote:192.168.221.201:40388 app_name:fwcp1-d1-fw-client01 app_public_addr:fwcp1-d1-fw-client01.app.xxx.admin app_uri:https://10.125.14.31/ng cluster_name:app.xxx.admin code:T2007I ei:0 event:app.session.start namespace:default public_addr:fwcp1-d1-fw-client01.app.xxx.admin server_id:bf4a4dd1-21f6-4c85-a43f-720f529b76b6 sid:540f9375-2c6c-4070-aa8a-721f7c20384d time:2022-10-24T09:41:09.619Z uid:e961b5f0-3baa-4faf-8da7-9e09b9d68786 user:admin events/emitter.go:263
2022-10-24T11:41:09+02:00 DEBU [PROXY:SER] Dialing from: "@web-proxy" to: "@local-node". trace.fields:map[cluster:app.xxx.admin] reversetunnel/localsite.go:199
2022-10-24T11:41:09+02:00 DEBU [PROXY:SER] Tunnel dialing to bf4a4dd1-21f6-4c85-a43f-720f529b76b6.app.xxx.admin. trace.fields:map[cluster:app.xxx.admin] reversetunnel/localsite.go:319
2022-10-24T11:41:09+02:00 DEBU [PROXY:SER] Connecting to 10.128.40.13:32892 through tunnel. trace.fields:map[cluster:app.xxx.admin] reversetunnel/localsite.go:598
2022-10-24T11:41:09+02:00 DEBU [PROXY:AGE] Transport request: teleport-transport. leaseID:1 target:app.xxx.admin:443 cluster:app.xxx.admin reversetunnel/agent.go:569
2022-10-24T11:41:09+02:00 DEBU [PROXY:AGE] Received out-of-band proxy transport request for @local-node [bf4a4dd1-21f6-4c85-a43f-720f529b76b6.app.xxx.admin]. cluster:app.xxx.admin reversetunnel/transport.go:206
2022-10-24T11:41:09+02:00 DEBU [PROXY:AGE] Handing off connection to a local "app" service. cluster:app.xxx.admin reversetunnel/transport.go:281
2022-10-24T11:41:09+02:00 DEBU [PROXY:SER] Succeeded dialing from: "@web-proxy" to: "@local-node". trace.fields:map[cluster:app.xxx.admin] reversetunnel/localsite.go:205
2022-10-24T11:41:09+02:00 DEBU [PROXY:SER] Dialing from: "@web-proxy" to: "@local-node". trace.fields:map[cluster:app.xxx.admin] reversetunnel/localsite.go:199
2022-10-24T11:41:09+02:00 DEBU [PROXY:SER] Tunnel dialing to bf4a4dd1-21f6-4c85-a43f-720f529b76b6.app.xxx.admin. trace.fields:map[cluster:app.xxx.admin] reversetunnel/localsite.go:319
2022-10-24T11:41:09+02:00 DEBU [PROXY:SER] Connecting to 10.128.40.13:32892 through tunnel. trace.fields:map[cluster:app.xxx.admin] reversetunnel/localsite.go:598
2022-10-24T11:41:09+02:00 WARN [APP:SERVI] Failed to handle client connection. error:[
ERROR REPORT:
Original Error: *errors.errorString EOF
Stack Trace:
    /go/src/github.com/gravitational/teleport/lib/srv/app/server.go:731 github.com/gravitational/teleport/lib/srv/app.(*Server).getConnectionInfo
    /go/src/github.com/gravitational/teleport/lib/srv/app/server.go:599 github.com/gravitational/teleport/lib/srv/app.(*Server).handleConnection
    /go/src/github.com/gravitational/teleport/lib/srv/app/server.go:584 github.com/gravitational/teleport/lib/srv/app.(*Server).HandleConnection
    /go/src/github.com/gravitational/teleport/lib/reversetunnel/transport.go:282 github.com/gravitational/teleport/lib/reversetunnel.(*transport).start
    /go/src/github.com/gravitational/teleport/lib/reversetunnel/agent.go:580 github.com/gravitational/teleport/lib/reversetunnel.(*agent).handleDrainChannels.func2
    /opt/go/src/runtime/asm_amd64.s:1571 runtime.goexit
User Message: TLS handshake failed
    EOF] app/server.go:585
2022-10-24T11:41:09+02:00 DEBU [PROXY:AGE] Transport request: teleport-transport. leaseID:1 target:app.xxx.admin:443 cluster:app.xxx.admin reversetunnel/agent.go:569
2022-10-24T11:41:09+02:00 WARN [APP:SERVI] Failed to close client connection. error:[
ERROR REPORT:
Original Error: trace.aggregate EOF
Stack Trace:
    /go/src/github.com/gravitational/teleport/api/utils/sshutils/chconn.go:113 github.com/gravitational/teleport/api/utils/sshutils.(*ChConn).Close
    /go/src/github.com/gravitational/teleport/lib/srv/app/server.go:586 github.com/gravitational/teleport/lib/srv/app.(*Server).HandleConnection
    /go/src/github.com/gravitational/teleport/lib/reversetunnel/transport.go:282 github.com/gravitational/teleport/lib/reversetunnel.(*transport).start
    /go/src/github.com/gravitational/teleport/lib/reversetunnel/agent.go:580 github.com/gravitational/teleport/lib/reversetunnel.(*agent).handleDrainChannels.func2
    /opt/go/src/runtime/asm_amd64.s:1571 runtime.goexit
User Message: EOF] app/server.go:587
2022-10-24T11:41:09+02:00 DEBU [PROXY:AGE] Received out-of-band proxy transport request for @local-node [bf4a4dd1-21f6-4c85-a43f-720f529b76b6.app.xxx.admin]. cluster:app.xxx.admin reversetunnel/transport.go:206
2022-10-24T11:41:09+02:00 DEBU [PROXY:AGE] Handing off connection to a local "app" service. cluster:app.xxx.admin reversetunnel/transport.go:281
2022-10-24T11:41:09+02:00 DEBU [PROXY:SER] Succeeded dialing from: "@web-proxy" to: "@local-node". trace.fields:map[cluster:app.xxx.admin] reversetunnel/localsite.go:205
2022-10-24T11:41:09+02:00 DEBU [AUTH]      ClientCertPool -> cert(app.xxx.admin issued by app.xxx.admin:264072570739875274449628954669027062524) auth/middleware.go:673
2022-10-24T11:41:09+02:00 DEBU [AUTH]      ClientCertPool -> cert(app.xxx.admin issued by app.xxx.admin:178967818217414070879805098676346956657) auth/middleware.go:673
2022-10-24T11:41:09+02:00 DEBU [RBAC]      Access to app "fwcp1-d1-fw-client01" granted, allow rule in role "access" matched. services/role.go:2053
2022-10-24T11:41:09+02:00 DEBU [APP:SERVI] Created app session chunk 8c1f6143-9e2c-4a36-99fa-1f8f1dae3d2d app/session.go:105
2022-10-24T11:41:09+02:00 DEBU [APP:SERVI] Creating tracker for session chunk 8c1f6143-9e2c-4a36-99fa-1f8f1dae3d2d app/session.go:358
2022-10-24T11:41:09+02:00 DEBU [APP:SERVI] Using async streamer for session chunk 8c1f6143-9e2c-4a36-99fa-1f8f1dae3d2d. app/session.go:328
2022-10-24T11:41:09+02:00 INFO [AUDIT]     app.session.chunk app_name:fwcp1-d1-fw-client01 app_public_addr:fwcp1-d1-fw-client01.app.xxx.admin app_uri:https://10.125.14.31/ng cluster_name:app.xxx.admin code:T2008I ei:0 event:app.session.chunk namespace:default server_id:bf4a4dd1-21f6-4c85-a43f-720f529b76b6 session_chunk_id:8c1f6143-9e2c-4a36-99fa-1f8f1dae3d2d sid:540f9375-2c6c-4070-aa8a-721f7c20384d time:2022-10-24T09:41:09.784Z uid:4e3eb69c-07c8-4bde-b1e3-ab9050d45ce5 user:admin events/emitter.go:263
2022-10-24T11:41:09+02:00 INFO             Round trip: GET /, code: 302, duration: 11.682µs tls:version: 304, tls:resume:false, tls:csuite:1301, tls:server:6170702e73626d2e61646d696e.teleport.cluster.local forward/fwd.go:187
2022-10-24T11:41:09+02:00 INFO [APP:WEB]   Round trip: GET /, code: 302, duration: 11.415251ms tls:version: 304, tls:resume:false, tls:csuite:1301, tls:server:fwcp1-d1-fw-client01.app.xxx.admin forward/fwd.go:187
2022-10-24T11:41:09+02:00 DEBU [RBAC]      Access to app "fwcp1-d1-fw-client01" granted, allow rule in role "access" matched. services/role.go:2053
2022-10-24T11:41:39+02:00 ERRO             Error forwarding to /ng, err: dial tcp 10.125.14.31:443: i/o timeout forward/fwd.go:181
2022-10-24T11:41:39+02:00 INFO [APP:WEB]   Round trip: GET /ng, code: 500, duration: 30.001794011s tls:version: 304, tls:resume:false, tls:csuite:1301, tls:server:fwcp1-d1-fw-client01.app.xxx.admin forward/fwd.go:187

I have made packets capture showing no problem. I don't know where to look to get this working.

Thanks

Raphaël

webvictim commented 1 year ago

The underlying issue here appears to be one of connectivity:

err: dial tcp 10.125.14.31:443: i/o timeout

I'd wager that there's a firewall rule not opened allowing the IP of the machine where the Teleport agent is running to access 10.125.14.31:443.

raphaelbkt commented 1 year ago

Hi,

Yes there was a network problem accessing the backend certificate. Now it is ok.

Thanks

Raphaël

louigitech commented 10 months ago

Hello @raphaelbkt ,

Did you manage to make it work with Teleport v14.x.x and FortiOS 7.2.x ?

Here is my configuration in the yaml file:

- name: "firewallxxx"
    uri: "https://XXX.XXX.XXX.XXX:4443/"
    insecure_skip_verify: true

When I launch the ressource "firewallxxx", I arrive to login page:

image

And then, when I login, I got a blank page yith this error "blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource":

image

From what I've understood, it's an header issue or something like this but didn't manage to find anything on the web...

Thank you !

raphaelbkt commented 10 months ago

Hi,

No I tested all configuration possible with no luck. I also had problems accessing Vmware vCenter and ESX servers via Teleport.

We had taken professionnal support and I opened this problem as a case. They told me that FortiOS was removing cookies on the answer but cookies are not working like that.

When we make a ssh tunnel to the fortinet it is working. So it is likely that Teleport did not implement a feature as it should.

In fact we had chosen Teleport pro to use http(s) proxy and we changed our minds as multiple applications did not work.

I must mention that half of the others applications (the most simples) did work well throught Teleport, including other firewall GUI like Palo Alto and pfsense.

In fact I understood later that this functionnality is developped in mind to put Teleport in front of your application, so you can make changes if needed.

Thanks

louigitech commented 10 months ago

Hi,

No I tested all configuration possible with no luck. I also had problems accessing Vmware vCenter and ESX servers via Teleport.

We had taken professionnal support and I opened this problem as a case. They told me that FortiOS was removing cookies on the answer but cookies are not working like that.

When we make a ssh tunnel to the fortinet it is working. So it is likely that Teleport did not implement a feature as it should.

In fact we had chosen Teleport pro to use http(s) proxy and we changed our minds as multiple applications did not work.

I must mention that half of the others applications (the most simples) did work well throught Teleport, including other firewall GUI like Palo Alto and pfsense.

In fact I understood later that this functionnality is developped in mind to put Teleport in front of your application, so you can make changes if needed.

Thanks

Thank you for your fast reply! I'll try to raise this case to Fortinet and see if they have an idea...

For VMware, have you tried this (didn't tried it yet): https://github.com/gravitational/teleport/discussions/25746

Again, thank you!

raphaelbkt commented 10 months ago

We have stopped using Teleport, but I am happy that solutions arise.

The Fortinet ticket was #8078052 and one of the problems I faced is that they asked 3 captures files:

The client to Teleport one is not possible because it is httpS only. I could capture ciphered traffic but couldn't decipher it for the purpose of the debug case.