auth login process - Listen and network binding

[BenPhegan]

Hi,

As discussed in https://github.com/databricks/cli/pull/560 I was looking to be able to run the databricks auth login process from a Docker container. During the attempt I came across an issue with the way the code is binding a listener for the callback.

It appears that the callback is hard coded to localhost:8200, meaning that the browser needs to be able to communicate with the databricks process on localhost:8200 when the authentication is complete. However, even when using -p 8200:8200 with a docker container the databricks application was only binding a private network port that could not easily be made to tunnel through to the host. This is probably not the only failure state with multi-interface networking. The code in question is here: https://github.com/databricks/cli/blob/main/libs/auth/oauth.go#L155 and the API description is here: https://pkg.go.dev/net#Listen

As stated in the documentation:

The address can use a host name, but this is not recommended, because it will create a listener for at most one of the host's IP addresses.

@pietern stated a preference for non-public interfaces only. There are a few approaches that could be taken to fix this that I could think of:

1. Looping through all interfaces, and explicitly bind to the set that reflect private IP address ranges.
2. Adding an --insecure-bind or equivalent flag. Default to the existing behaviour, but allow this flag to bind to all network interfaces by changing from localhost:8200 to only :8200 as per the fix described in #560 .
3. Leave this as a known constraint and force people to address through the use of Docker magic. There is probably a way to do this using host networking (only Linux) or with some other docker networking setup (not fully investigated as yet). The key takeaway is that this will not work out of the box as people would expect.

There are probably other approaches as well. Happy to make the changes, but it would be good to have a discussion around the best approach first.

Regards,

Ben

[BenPhegan]

@andrewnester - anything I can progress on this? Any thoughts as to the best approach?

[Uru9V]

hey @BenPhegan I also had trouble to use the new CLI in a docker container (bitbucket pipeline). I tried several authentication methods Databricks provided on their website but no of them was valid when I used the CLI in a bitbucket pipeline with a custom docker image or with Docker Desktop. Because of the failed auth process I wasn't able to use any of the API calls over the CLI.

Did you guys already found out a workaround for this (for e.g. which configuration I have to set for the container to get this CLI work)? Or any plans to get this work out-of-the-box for docker container in the future?

For the moment I replaced the CLI by requesting the API directly in the bitbucket pipeline but for a better readability and less code lines in my pipeline I would be happy when you guys could get this work in the future.

Thanks in advance.

Best regards, David

[BenPhegan]

Hi @Uru9V , we got it working with a combination of a patched version of the CLI (using this: https://github.com/databricks/cli/pull/560/files#diff-c7f46e48b763b5ba29837574739bacc7a9eb04d1f5a67e29c7496171c8a0a7a2) and this PR: https://github.com/databricks/cli/pull/561

I would love to see fixes in so that we can use Docker interchangeably with the CLI binary natively. Waiting currently on discussions with maintainers as to which way they would like to see this fixed.

[andrewnester]

@BenPhegan Hi Ben! Sorry for the late response and thanks for the provided examples. If I see it correctly, binding to all :8020 instead of localhost:8020 seems to help the issue, correct? I was curious of 2 things:

1. If CLI is running inside the docker container, does oauth callback works correctly and show browser page in your case?
2. if CLI is running not inside the container, is oauth still working?

I'd expect answer for 1 is yes, but 2 might be no?