Couldn't connect to Prefect Server at http://localhost:4200/graphql

[jackhawa]

Hello, enjoying the MLOps ecosystem. Great work!

I am following the minikube set up and was able to operate mlflow and jupyter hub but could not connect to graphql endpoint from prefect with the error: "Couldn't connect to Prefect Server at http://localhost:4200/graphql"

I have made sure that I am using install_locally=True, aws=False and the graphql site at http:localhost:4200 is up and running.

I have also executed this command to forward the graphql 4200 port:

kubectl port-forward -n prefect svc/prefect-server-apollo 4200

But for some reason it is still not connecting.

Any tips to troublehsoot?

[jackhawa]

Here is the list of kubernetes services in case it helps

[bernardolk]

Hello there. Does the prefect server graphql pod contain any useful logs? Or even any other prefect pod, for that matter? There isn't much info here to work with yet but I would recommend that you try using the graphql endpoint from a python script using the prefect client just to make sure you can execute graphql queries against it (You can checkout prefect documentation on how to use the prefect client for graphql querying). If you can, then the problem may be with the prefect UI pod. It may be not configured correctly to reach the graphql service.

[jackhawa]

It turns out there is an error in prefect-server-agent.

It is trying to connect to /graphql url without any hostname attached to it. I am assuming that is the error. If so I am not sure why it is dropping the hostname http://localhost:4200 since I have added install_locally to my terraform config file.

[jackhawa]

Locally I am able to reach out and query graphql api from a python script.

[bernardolk]

Can you check the env vars in the prefect server agent pod?

kubectl describe pod -n prefect prefect-server-agent-<somerandomid>

[bernardolk]

Check if you have PREFECT__CLOUD__API set to something

[bernardolk]

Also, do the same with the UI pod and check the PREFECT_SERVER__APOLLO_URL env var value

[jackhawa]

PREFECT_CLOUD_API is pointing to http://prefect-server-apollo.prefect:4200/graphql PREFECT_SERVER__APOLLO_URL points to http://localhost:4200/graphql

[jackhawa]

Would you have any idea where to go from here? Any tips is appreciate. Thank you in advance.

[bernardolk]

Hmm, these look ok. Considering you can reach the graphql endpoint by port-forwarding, it really should be working. I would recommend you open an issue with prefect, I am out of ideas since the variables look ok :(

[jackhawa]

Ok thanks for your help!