Unable to connect from a Rust application

[ramnivas]

I am trying to deploy a Rust application that uses a Railway Postgres service. I cannot connect to the Postgres service from my Rust app (running on my laptop) using either OpenSSL or Rustls (with native certificates).

Here is a reproducible example: https://github.com/ramnivas/railway-postgres-rust

Here is the error I get, which, I believe, points to the SSL certificate provisioned by this service.

With Rustls (with native certs):

Error { kind: Tls, cause: Some(Custom { kind: InvalidData, error: InvalidCertificate(UnknownIssuer) }) }

With OpenSSL:

Error { kind: Tls, cause: Some(ConnectError { error: Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 369098857, library: "STORE routines", function: "ossl_store_get0_loader_int", reason: "unregistered scheme", file: "crypto/store/store_register.c", line: 237, data: "scheme=file" }, Error { code: 2147483650, library: "system library", function: "file_open", file: "providers/implementations/storemgmt/file_store.c", line: 267, data: "calling stat(/nix/store/nii4qs05l95b6lva5sr75kslc0pdrmym-openssl-3.0.9/etc/ssl/certs)" }, Error { code: 369098857, library: "STORE routines", function: "ossl_store_get0_loader_int", reason: "unregistered scheme", file: "crypto/store/store_register.c", line: 237, data: "scheme=file" }, Error { code: 2147483650, library: "system library", function: "file_open", file: "providers/implementations/storemgmt/file_store.c", line: 267, data: "calling stat(/nix/store/nii4qs05l95b6lva5sr75kslc0pdrmym-openssl-3.0.9/etc/ssl/certs)" }, Error { code: 369098857, library: "STORE routines", function: "ossl_store_get0_loader_int", reason: "unregistered scheme", file: "crypto/store/store_register.c", line: 237, data: "scheme=file" }, Error { code: 2147483650, library: "system library", function: "file_open", file: "providers/implementations/storemgmt/file_store.c", line: 267, data: "calling stat(/nix/store/nii4qs05l95b6lva5sr75kslc0pdrmym-openssl-3.0.9/etc/ssl/certs)" }, Error { code: 167772294, library: "SSL routines", function: "tls_post_process_server_certificate", reason: "certificate verify failed", file: "ssl/statem/statem_clnt.c", line: 1890 }]))) }, verify_result: X509VerifyResult { code: 19, error: "self-signed certificate in certificate chain" } }) }

[ramnivas]

Also note, if I add a query parameter sslmode=disable, the connection is established, but of course, that doesn't seem the right way to go,

[joggienl]

If you are using the default Postgresql service from Railway, it uses a self signed certificate. It might be that Rust is not allowing that by default, so there might be an option to allow it.

I do not know how you would do that in Rust, for javascript/typescript when using pg there is an option called rejectUnauthorized which can be set to false.

Hope this helps!

[ramnivas]

@joggienl The equivalent option in Rust is the?sslmode=disable query parameter, which works, but seems unsafe.

[joggienl]

@ramnivas, no that is not the same. The rejectUnauthorized is meant to manually "trust" unsafe certificate while still using SSL. The sslmode param is simply requesting to totally disable SSL, which indeed is unsafe.

Trusting unsafe certificates is obviously something you should only do if you know where you are connecting to.

*edit: I hope this difference makes a bit sense. The rejectUnauthorized is handled by the client, the sslmode is requesting the server to allow non-ssl connections. So the rejectUnauthorized is something you configure your client with and that is why sslmode is added like a queryparam because it is being sent to the server.

[ramnivas]

Good point. I guess a better equivalent would be to implement a custom verifier (for example, in Rustls https://docs.rs/rustls/latest/rustls/client/trait.ServerCertVerifier.html). But the core issue remains, since there may not be a good way to safely verify such self-signed certificate.

[joggienl]

I do not know what to advise here, in general you should be able to have an option to either ignore SSL errors on the client or have an option more specifically for self signed certificates. Hope you'll find that out!

[ramnivas]

Thanks for your help.

As far as I see, there is no good solution to use this service from Rust. Ignoring SSL errors in client can be managed through some code, but carries the risk of MTM attacks.

A solution would be to change this service along the lines of Yugabyte [1][2] by publishing a PEM file. Then I could add builder.set_ca_file("database_cert.pem") for a proper validation. But maintainers of this services would need to decide it that is something they want to pursue.

[1] https://docs.yugabyte.com/preview/drivers-orms/orms/nodejs/ysql-prisma/#specify-the-configuration [2] https://docs.yugabyte.com/preview/yugabyte-cloud/cloud-secure-clusters/cloud-authentication/#download-your-cluster-certificate

[joggienl]

Having a valid certificate on the service side is imo the best solution, I think that is what you are implying here. Another solution for your specific use case could be to custom add Postgresql to your project and only allow access via the private network (you can do that by removing the DATABASE_URL environment variable and removing the TCP proxy)

[ramnivas]

@joggienl

Having a valid certificate on the service side is imo the best solution, I think that is what you are implying here.

Exactly.

I am giving the private network a try by using g $DATABASE_PRIVATE_URL with sslmode=disable, which seems safe since the private network isn't available for anyone to introduce a MITM attack (I am running into issues with my app, unable to resolve the hostname, but that is a separate use).

However, this still exposes public usage to MITM (my apps setup requires some operations from outside the server).

[ramnivas]

@joggienl For context: I am trying to deploy Exograph to Railway with an eye on providing a template to get a nice "one-click" deployment experience.

With private networking (assuming that I can overcome the host resolution issue), I think the server->database will be okay. However, certain usages, such as migrations and schema verification, require access from developer machines or other non-project machines. This is where I am uncomfortable with the use of public database URL

[joggienl]

Hey @ramnivas , looks nice Exograph, haven't looked into that one. Saved to (long) list!

About the network issues: there is a small delay in which the private network becomes available (about 100ms I believe). It is described in the Railway docs. You could try to add ?connect_timeout=300 to the postgresql connection string (if that is where the problem is). That might help.

Another caveat is that if you are using Alpine images, you need to add ENABLE_ALPINE_PRIVATE_NETWORKING=true to the environment variables of those images see the docs here.

As for the need for public access for setup, maybe you can add a container to the project that does this setup? You could custom build a docker image to pull that off, and maybe some docs that describe that it can be deleted afterwards (if necessary). Just a thought, there might be other or even better ways to do it ;-)

[melissa-hale]

I wonder if this issue is related to this community feedback post: https://community.railway.app/feedback/generate-v3-x-v509-certs-for-postgres-te-370861a1

I'd love to make time for implementing improvements to our postres images soon™️, and this is on my radar. In the meantime, I'm always happy to review PRs.

[ramnivas]

@melissa-hale I don't think they are directly related since the issue exists even with OpenSSL. The core issue here is that Railway Postgres uses a self-signed certificate, so clients have no real way to connect over SSL.

[melissa-hale]

Okay, thanks @ramnivas. Will keep this in mind. Not sure of any timeline, but will keep you posted on our decisions. I know you were trying to build a template for Exograph, are you blocked because of this?

[ramnivas]

Thanks @melissa-hale.

I am not quite blocked (https://exograph.dev/docs/deployment/railway), but I had to go for a sub-optimal solution:

• Use private networking during execution, which is fine since the private network is restricted with no possibility of MTM
• Use disabled SSL during migration (which is carried out at the build time; hence, no private networking is available). This is problematic (and not just for Exograph; any connection made from outside the project, for example, using psql will have the same issue).

As for the template, I am blocked on a different issue (there is no way to express dependency between components). For example, I want to start Exograph only after the Postgres service is up and running (there are workarounds, such as using nc to keep waiting for Postgres to accept connection, but those aren't elegant).

[melissa-hale]

Wow this is great feedback @ramnivas , thanks for taking the time to type it out.

• No private networking at build time - interestingly, I think this is the first time I've seen this raised, but I can see that it would be nice especially for database migrations. There are some things we're exploring this quarter that I think make this possible? For now though, I've made a note of it.
• No way to express dependencies - yes, completely agree! I've captured the feedback in our Forums here and will also raise it in one of our related project channels.

That leaves the issue with the self-signed cert which we can continue tracking here.

[ramnivas]

Happy to be of help, @melissa-hale. I will keep an eye on changes in these areas.

[brody192]

Hey all, this was fixed in PR #15

Once new images are published all you would need to do is redeploy, the added logic will detect the invalid certificate and regenerate it for you.

Two community members have confirmed the changes made in PR 15 worked for them with rustls.

[brody192]

Additionally, Railway now supports private networking during build when you enable the new builder in your service settings!