How can I clone a repo located in LAN?

[drafish]

Clone Git Repository only works for WAN. It's using a cors-proxy server to solve cors problem. Here is the server address

https://corsproxy.remixproject.org/

It can't access LAN, so I can't clone a repo located in LAN.

I'v already tried to start a cors-proxy server and Remix IDE on my computer, and replace the cors-proxy server address with my local address.

http://localhost:9999/

I'm using isomorphic-git/cors-proxy.

I installed it like this

npm install -g @isomorphic-git/cors-proxy

and start it like this

cors-proxy start

But it didn't work for LAN or WAN. I can't clone any repository. I didn't see any errors, just pending like this

Can you guys give me some details about how to deploy a cors-proxy server?

@ryestew @Aniket-Engg @yann300 @LianaHus @bunsenstraat @ioedeveloper

[bunsenstraat]

Let's discuss what you want to achieve @drafish as the cors proxy handles cors issues when talking to GitHub etc. Where is the repo you are cloning located?

[drafish]

The repo located in the LAN (local area network) of my company. I want to deploy a cors proxy in LAN, so that I can clone the repo located in LAN. @bunsenstraat

[bunsenstraat]

Are you running a local git server? The cors proxy is only used because GitHub doesn't allow web clients to fetch data in the way we do it. If you tell me a bit about your setup I can run a test and see if we can use cloning other sources. I'm assuming there is some security setup and so on. Our setup has only been tested on GitHub and gitlab

[drafish]

Yeah, some thing like local. The git server is deployed in the LAN of my company which can't be visited through WAN. I totally understand why the cors proxy of remix didn't work, because it can't access LAN. The cors proxy of remix only works for git server which has public ip, like github and gitlab.

If I need to clone a repo located in LAN, I should use my own cors proxy which also located in the same LAN. There is no way to configure the cors proxy address, so I also need to replace it in code and start my own remix service. It will be much better if the cors proxy address is configurable. That's just a suggestion. What I'm concerned now is how to deploy a cors proxy.

I don't know what kind of setup you're talking about. If you want to run a test, just tried a repo not located in the same LAN as the cors proxy, and you will see the problem. @bunsenstraat

[bunsenstraat]

You don't need a cors proxy if your server doesn't refuse the request. But as you said our cors proxy server can't access your lan. The obvious option is to disable the cors config in remix and just let the client contact your lan directly. The dgitprovider class has the config, you can comment out the proxy option. There is no need for it if your server accepts the request.

[drafish]

I see your point. If I can get the git server support cors access, problem will be solved.

The git server we're using now is bitbucket. I'm not sure it support cors or not. I'v searched the answer for hours, still not get it. Someone asked a similar question in the bitbucket community, see the link below.

https://community.atlassian.com/t5/Bitbucket-questions/Access-Bitbucket-API-from-Client-Browser-CORS-access-support/qaq-p/1039301

He didn't get the answer either.

Can you please tell me how did you deploy the cors proxy? I still think it's necessary for people to use their own cors proxy. Not just for the LAN problem. Now, everyone have to use remix's cors proxy. Is it too centralized? Github and Gitlab both support private repository. I will feel much comfortable, if I can use my own cors proxy when I'm cloning a private repository. @bunsenstraat

[bunsenstraat]

I see your point too. Up to now I haven't been able to access bitbucket from remix. The cors proxy doesn't help. Gitlab yes, for cloning public repos, not private. Only GitHub works fully. So running this proxy wouldn't help. I will run some more tests. The safest way to go for you there is to use remixd or remix desktop. Either one. You'd be doing your git operations locally and not in remix. But I will do some more research. Running this cors proxy is simple in itself. You just run the npm package. We do it in a service. Nothing special about it.

[drafish]

Hey @bunsenstraat I finally get the cors proxy work. Seems like the cli usage of @isomorphic-git/cors-proxy not work properly. However the middleware usage works. Just like you said, do it in a service.

I know why gitlab not works fully. Check the link below. https://github.com/ethereum/remix-project/blob/master/apps/remix-ide/src/app/files/dgitProvider.js#L68-L69

username and password not set properly. github and gitlab are different.

Here is the docs link. https://isomorphic-git.org/docs/en/onAuth#oauth2-tokens

I modified the code. It works for gitlab.

You guys may need to figure out a way to make the OAuth2 Tokens adapt to more git services.

I also tested on bitbucket with correct OAuth2 Tokens. But it didn't work. Still don't know why.

[bunsenstraat]

I can't get the gitlab to work myself... but I will give a try again.

[bunsenstraat]

Might have to go and use a customised proxy.. working on that now.

[drafish]

Hey @bunsenstraat I finally get the cors proxy work. Seems like the cli usage of @isomorphic-git/cors-proxy not work properly. However the middleware usage works. Just like you said, do it in a service.

I know why gitlab not works fully. Check the link below. https://github.com/ethereum/remix-project/blob/master/apps/remix-ide/src/app/files/dgitProvider.js#L68-L69

username and password not set properly. github and gitlab are different.

Here is the docs link. https://isomorphic-git.org/docs/en/onAuth#oauth2-tokens

I modified the code. It works for gitlab.

You guys may need to figure out a way to make the OAuth2 Tokens adapt to more git services.

I also tested on bitbucket with correct OAuth2 Tokens. But it didn't work. Still don't know why.

Hi @bunsenstraat , I got the screenshot wrong. I corrected it already. Sorry about that.

And I figure out why bitbucket didn't work. Check the function below, located in remix-project/node_modules/isomorphic-git/index.cjs line 6243 ~ 6245

This is not right for bitbucket. Check the docs link below. https://confluence.atlassian.com/bitbucketserver0516/personal-access-tokens-966061199.html

Only need one parameter, which is the token. In my case, token is password. So I modified the function like this.

function calculateBasicAuthHeader({ username = '', password = '' }) {
  return `Bearer ${password}`
}

If the token is username, then it should be

function calculateBasicAuthHeader({ username = '', password = '' }) {
  return `Bearer ${username}`
}

I tested it on the private bitbucket service located in the LAN of my company. It worked.

I also tested on https://bitbucket.org. That didn't work. Still don't know why.

[bunsenstraat]

That's really insightful! It sees like you can provide your own headers

https://github.com/isomorphic-git/isomorphic-git/blob/9b183f13a381a0134b83e4d211310fb1872ffff4/src/managers/GitRemoteHTTP.js#LL19C1-L28C2

const updateHeaders = (headers, auth) => {
  // Update the basic auth header
  if (auth.username || auth.password) {
    headers.Authorization = calculateBasicAuthHeader(auth)
  }
  // but any manually provided headers take precedence
  if (auth.headers) {
    Object.assign(headers, auth.headers)
  }
}

I will some changes in our code to deal with this situation. And see what we can do with this. Super!