How to use apm under the WALL

[lijunle]

The story

As you may know, there is a WALL in China. Many websites are affected by this great wall, including npm, Amazon S3. As a result, there is something like Taobao NPM mirror to resolve problems.

While using apm, I feel sad. The tool only output one line when installing a package. Under the wall, it always says, installing... and hangs. Half or one hour passed, failed. :(

The tool even has no options to out more log to debug what's going wrong!

That is why I want to draft a document to help the user to across the wall to install a package.

Note: I just modify the generated .js file in the Atom installed directory. Maybe you need a backup before start. This tool also cover Windows users.

Debug

First of all, we need to output more things to be aware what is happening under one line.

Open Atom/resources/app/apm/node_modules/atom-package-manager/lib/install.js

1. Add process.env.NODE_DEBUG = 'request'; after the require statements. (Reference)

2. Inside Install.prototype.installNode function, pass { streaming: true } in _this.fork function. Like this:

   // around line 68
   return _this.fork(_this.atomNodeGypPath, installNodeArgs, {
    env: env,
    streaming: true, // <--- add this line
    cwd: _this.atomDirectory
   }, function(code, stderr, stdout) {
    ...
   }

3. Do the same thing in the fork invoke in Install.prototype.installModule function.

   // around line 140
   installOptions = {
    env: env,
    streaming: true <--- add this line
   };
   ...
   return this.fork(this.atomNpmPath, installArgs, installOptions, (function(_this) {
    ...
   }

   Add streaming: true parameter will let the sub-process log output to console directly. (Reference)

OK, done. Now, run apm install <package> in your console, you get more logs than before. When the output hangs, investigate which URL is requesting.

URLs

There are mainly three sources will be requested when install a package.

1. Atom API. The request to this API happens at the beginning, to get package metadata.
2. Amazon S3. This URL is affected by the wall strongly. All atom packages are stored in Amazon S3. After the metadata is returned, apm starts to download the package from S3.
3. NPM. When there are lots of request, this API will be blocked. After the Atom package is downloaded, apm will request its dependencies from NPM store.

   Configurations

Note: once you modify the configuration, run apm install <package> and check its log. The configuration may not work. (You know, the wall is great.) Then, continue modify your configuration and re-check.

There are two ways to configure apm.

1. .apmrc file. The global .apmrc file exists in Atom/resources/app/apm/node_modules/atom-package-manager/, and the user specified is in $HOME.
2. Environment variable. Some configurations are only exposed to environment. Check process.env in config.js.

Basically, add the following two lines in .apmrc file will set proxy. (Reference). But due to some issues, such proxy does not work.

proxy = http://HOST:PORT
strict-ssl = false

As a solution, run the apm command, then check which is the cause of the failure. Then add workaround.

1. Atom API. When blocked when request Atom API. Add a environment variable ATOM_API_URL to point to your reverse proxy. (Reference) (About reverse proxy, read it below.)
2. Amazon S3. Environment variable ATOM_NODE_URL points to your reverse proxy. (Reference)

3. NPM. Thanks to the cool Taobao NPM. They proxy the NPM packages for us. add the following to .apmrc file:

   registry = https://registry.npm.taobao.org

   Reverse Proxy

If you do not have a proxy, you can use mine:

ATOM_API_URL = http://atom.lijunle.tk/api
ATOM_NODE_URL = http://atom.lijunle.tk/dist

It is recommended to use nginx to set up your own reverse proxy. Sample configuration:

server {
    listen 80;
    server_name atom.lijunle.tk;

    location /dist/ {
        proxy_pass https://gh-contractor-zcbenz.s3.amazonaws.com/atom-shell/dist/;
    }

    location /api/ {
        proxy_pass https://atom.io/api/;
    }
}

[kevinsawicki]

Thanks for investigating this and writing it up, I think adding a doc in this repository that is linked from the README would be great.

I'll also fine adding some command line options to make the streaming changes and the debug changes you've mentioned, that way people can just run apm debug-install or something to check their connections and do a test install of a simple package.

[lijunle]

@kevinnathan that is great! At first, I could like to use something like this apm install --verbose to get more output, but it does not work.

Besides, is there a doc to describe how to use my develop apm instead of the installed one? I want to make some changes to the source code, but not find the way. :-(

[kevinsawicki]

You should be able to just clone this repo and build is using grunt, Atom itself just shells out to apm so testing it from the command line on a locally cloned version is probably your best option.

[olingerc]

I've tried to use your workaround to get past our company proxy whicn does not like https connections. Using your reverse proxy to access the api works since I can correctly search for packages, but as soon as apm tries to download a package, it tries to do so from https://www.atom.io/api/ even with ATOM_NODE_URL set.

ATOM_API_URL=http://atom.lijunle.tk/api apm search atom-beautif

Works nicely but

ATOM_API_URL=http://atom.lijunle.tk/api ATOM_NODE_URL=http://atom.lijunle.tk/api apm install atom-beautif

gives the following error

Installing atom-beautify to /home/christophe/.atom/packages Unable to download https://www.atom.io/api/packages/atom-beautify/versions/0.13.2/tarball: read ECONNRESET

I opened a separate issue: https://github.com/atom/apm/issues/197 for this but was hoping you maybe had an idea?

[yansern]

I'm working behind a corporate firewall where everything needs to goes through a corporate proxy server. However, any hostname that doesn't have a "." in it will be granted direct access.

Therefore, I needed to rewrite everything from "http://atom.io" to "http://atom_io", including npm registry's which is proxied via "http://atom_io/npm".

Doesn't matter if your situation is the same as mine. As long as you need to proxy atom's url, you can try the configuration stack I use below:

In the hosts file:

1.2.3.4 atom_io
# replace with the IP address where your nginx server is hosted

In the .apmrc file:

strict-ssl = false
registry = http://atom_io/npm/

In the environment variables:

ATOM_API_URL = http://atom_io/api
ATOM_NODE_URL = http://atom_io/dist

In the nginx configuration file:

server {
    listen 80;
    server_name atom_io;

    subs_filter_types application/json;
    subs_filter "https://www.atom.io" "http://atom_io";
    subs_filter "registry.npmjs.org" "atom_io/npm";

    proxy_redirect https://codeload.github.com http://atom_io/codeload.github.com;

    location /dist/ {
        proxy_pass https://gh-contractor-zcbenz.s3.amazonaws.com/atom-shell/dist/;
    }

    location /api/ {
        proxy_pass https://atom.io/api/;
    }

    location /npm/ {
        proxy_pass http://registry.npmjs.org/;
    }

    location /codeload.github.com/ {
        proxy_pass https://codeload.github.com/;
    }
}

I use the subs_filter directive which is available via the nginx-extras package on ubuntu. You can install using apt-get install nginx-extras.

Sometimes different apm packages may be connecting to different sites to download their respective packages. Use apm install --debug package-name-here to identify the urls they are trying to download from.

Once you have identified it:

• If you need to replace the URL in the http response content on your nginx proxy, use subs_filter.
• If you need to replace the URL in the http response header's location param on your nginx proxy, use proxy_redirect.

Thanks @lijunle for providing the initial guide.

[Loyalsoldier]

It seems that there is no "atom-package-manager" folder at all……

[Loyalsoldier]

Update: this method seems only work at CLI, not succeeds in atom.exe package install page

Tell you guys how to simply test and config the apm

The way of @lijunle seems pretty difficult because I can't find any folder called "atom-package-manager" in my Windows.

But I eventually find out the simpler way according to the way of @lijunle

My OS environment is like this:

• Installed Node.js
• Installed npm (highly recommend to use cnpm instead of npm)
• Installed Atom.

Now, let's rock!

1. Own your proxy first (recommend shadowsocks-gui, and use its PAC provided by GFWList)
2. Test if you can open Atom API
3. Test if you can download Atom with your full network speed, which at least satisfy you
4. If you can't open No.2 or No.3 is very slow, then edit the PAC of you shadowsocks-gui
5. Add the following domain into the PAC file then save it, like this:

"atom.io",
"s3.amazonaws.com",

The most important step,

use the npm mirror provided by Taobao by adding the following command into your .apmrc file, which can be located at ~./Users/YOURUSERNAME/.atom/.apm, then save:

registry = https://registry.npm.taobao.org

As far as I'm concerned, npm is the sin commonly. Enjoy it !

[fengmk2]

All atom packages are stored in Amazon S3

It's hard to mirror.

[Loyalsoldier]

@fengmk2 Use proxy.

[lijunle]

@Loyalsoldier for ss users, it is easy to enable global proxy, then apm install/upgrade, then disable global proxy after operation done. :smile_cat:

[Z-Clive]

@lijunle Thanks a lot! That WALL thing is really a pain in the a**!

[lijunle]

I want to mention that, this post is old and some information is not up-to-date. Apm have integrated some idea to itself, for example, the --verbose argument. And currently, the apm proxy works great in most of the case.

However, the worst issue I encountered recently is the speed of download atom - AWS is very slow in China without proxy. :disappointed:

[kigawas]

proxychains4 apm install xxx may help.

[ShadowRZ]

@lijunle Contact me with PGP.

[Loyalsoldier]

First, I think you'd better buy a fast VPN service or something like that(Recommend a Shadowsocks service).

(Only for Chinese)Then, you need a tool with black-list(against with white-list like gfwlist) and intelligent IP judgement function. I recommend to use MEOW. Because MEOW can automatically to detect where the IP address you are visiting belongs to and decide whether to use proxy or not.

Finally, you just set your OS wide proxy or PAC proxy, and add proxy settings to .bashrc or .zshrc configuration file. As a result, all of your command running in the Terminal will go through MEOW.

Everything is great!

One more thing: If you are Chinese, for better performance, you can use taobao npm registry for apm by adding the following line into ~/.atom/.apmrc configuration file:

registry=https://registry.npm.taobao.org/

[Loyalsoldier]

@lijunle Using proxy with white-list or intelligent IP judgement like MEOW to accelerate the apm download process.

White-list tools like Shadowsocks(using gfwlist by default) can't accelerate most domains and websites outside China.

[ShadowRZ]

Can we solve this problem from the source of this problem?

[ShadowRZ]

Tor?

[Gaohaoyang]

Another way you can try it.

First, search the plugin you want to install, such as atom-beautify. Get its git url.

Then

cd /Users/YOURNAME/.atom/packages
git clone git@github.com:Glavin001/atom-beautify.git
cd atom-beautify
npm install

Restart Atom, done!

[ShadowRZ]

It hints me. Maybe Atom plugins are released with GitHub Release.

于 2017年3月23日 GMT+0800AM11时29分41秒, "浩阳" notifications@github.com 写到:

Another way you can try it.

Search the plugin you want to install, such as atom-beautify. Get its git url.

Then

cd /Users/YOURNAME/.atom/packages
git clone git@github.com:Glavin001/atom-beautify.git
cd atom-beautify
npm install

Restart Atom, done!

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/atom/apm/issues/174#issuecomment-288607531

-- 使用 K-9 Mail 发送自我的手机的说～ 来自NicoNicoNi的HID_System的说～

[cniweb]

After change .apmrc in C:\Users\xxx\.atom\.apm and save, the file changes back to default settings after restart atom. :confused:

[mjysci]

I followed the steps set up a reverse proxy via nginx myself. And it turns out the API url https://atom.io/api/ no longer working. It's 404 if visit it directly and 502 Bad Gateway via reverse proxy.

Any idea what the new api url should be?

[maninalift]

@mjysci where are you running nginx? nginx returns a 502 when it fails to connect to the service that it is forwarding requests to. You will get this if the place where you are running nginx is also blocked.

[mjysci]

@maninalift Thanks for your reply! I'm runing nginx in a VPS locate in Ohio, USA. I'm sure it's not blocked. The problem is the API url https://atom.io/api/ is no longer working or move somewhere else. It's 404 and shows "THIS PAGE COULD NOT BE FOUND 404 DO NOT PANIC Perhaps you misspelled the url or it has been removed."
I'm wondering what's the new atom API url?

[wanancat]

@Loyalsoldier

1. Own your proxy first (recommend shadowsocks-gui, and use its PAC provided by GFWList)
2. Test if you can open Atom API
3. Test if you can download Atom with your full network speed, which at least satisfy you
4. If you can't open No.2 or No.3 is very slow, then edit the PAC of you shadowsocks-gui
5. Add the following domain into the PAC file then save it, like this:

I can open Atom and download it in normal speed by enable global proxy of Shadowsocks, but Atom API shown 404 same as @mjysci .

The page of Meow is 404 as well.

I add the taobao npm registry in ~/.atom/.apmrc, it doesn't work. apm install --check failed again.

Any advice? :-S