Is it possible to access the OpenSearch VPC endpoint from outside AWS?

[RGB314]

Hi,

I'm trying to access the OpenSearch VPC endpoint from outside AWS (a Java client running on my PC and another running on a server). I have obtained the correct credentials from software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider (which works with other AWS services) but I'm unable to connect. The code and the error are:

// Kotlin code
import io.github.acm19.aws.interceptor.http.AwsRequestSigningApacheV5Interceptor
import org.apache.hc.client5.http.classic.methods.HttpGet
import org.apache.hc.client5.http.config.RequestConfig
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient
import org.apache.hc.client5.http.impl.classic.HttpClients
import org.apache.hc.core5.http.ClassicHttpResponse
import org.apache.hc.core5.http.HttpHost
import org.apache.hc.core5.http.HttpRequestInterceptor
import org.apache.hc.core5.util.Timeout
import org.junit.jupiter.api.Test
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider
import software.amazon.awssdk.auth.signer.Aws4Signer
import software.amazon.awssdk.regions.Region
import software.amazon.awssdk.utils.IoUtils.toUtf8String
import kotlin.test.assertEquals

    @Test
    fun test() {

        val signer = Aws4Signer.create()
        val interceptor: HttpRequestInterceptor = AwsRequestSigningApacheV5Interceptor(
            "es",
            signer,
            ProfileCredentialsProvider.builder()
                .profileName("myprofile")
                .build(),
            Region.US_EAST_1
        )

        val config = RequestConfig.custom()
            .setConnectTimeout(Timeout.ofMilliseconds(5000L))
            .setConnectionRequestTimeout(Timeout.ofMilliseconds(5000L))
            .setResponseTimeout(Timeout.ofMilliseconds(5000L))
            .build()

//        val request = HttpGet("https://www.google.com")
        val request = HttpGet("https://mydomain.us-east-1.es.amazonaws.com")

        val httpClient: CloseableHttpClient = HttpClients.custom()
            .setDefaultRequestConfig(config)
            .setProxy(HttpHost.create("http://myproxy.net:8443"))
            .addRequestInterceptorLast(interceptor)
            .build()
        httpClient.execute(request) { response: ClassicHttpResponse ->
            assertEquals(200, response.code)
            println(toUtf8String(response.entity.content))
        }

    }

// Exception stacktrace
java.net.SocketTimeoutException: Read timed out

    at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:288)
    at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:314)
    at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355)
    at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:808)
    at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
    at org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:149)
    at org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
    at org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:247)
    at org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:54)
    at org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:299)
    at org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:175)
    at org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:218)
    at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$InternalConnectionEndpoint.execute(PoolingHttpClientConnectionManager.java:717)
    at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.execute(InternalExecRuntime.java:216)
    at org.apache.hc.client5.http.impl.classic.ConnectExec.createTunnelToTarget(ConnectExec.java:233)
    at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:151)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:113)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:152)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:116)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170)
    at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:245)
    at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:188)
    at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:162)

This is related to my attempt to connect to the cluster and be able to get raw search responses to reduce latency. See https://github.com/opensearch-project/opensearch-java/issues/905

[dblock]

This looks like a networking problem and not an interceptor one. Are you able to hit your endpoint with curl?

I'll close this issue, but we can reopen if you believe this is something that the interceptor code is causing.

[RGB314]

Commented line val request = HttpGet("https://www.google.com") works though.

Can't access via curl either - Access Denied. I'm inside my company's network and AWS is obviously outside and needs the proxy.

I just wanted to know if AWS VPC endpoints (any service, not just OS) are accessible from outside AWS at all (if there aren't any proxies involved)?

Trying to access locally in order to have a quicker turnaround instead of having to deploy to AWS using the company's build pipeline.

[dblock]

AFAIK a VPC that has a network policy to only allow access from the VPC then that's what it allows. Your options are AWS Direct Connect, AWS site-to-site VPN, or via AWS VPN Client. There's some good material about this in https://aws.amazon.com/blogs/big-data/network-connectivity-patterns-for-amazon-opensearch-serverless/. For temporary access an SSH tunnel can work, but it's more involved.