Open GoogleCodeExporter opened 8 years ago
As you probably know, GAE does not work like the other PAAS. GAE applications
have restrictions: 1GB Outgoing Bandwidth, 1GB Incoming Bandwidth, 5MB URLFetch
Request, 32MB URLFetch Response, ..
So when APJP needs to download something big (>32MB) with GAE, APJP will split
that download in multiple small downloads (HTTP Range Request).
But that does not work when you forward to another APJP_REMOTE server.
APJP_REMOTE will need to be updated.
Original comment by jeroen.v...@gmail.com
on 11 Mar 2013 at 6:31
[deleted comment]
[deleted comment]
[deleted comment]
[deleted comment]
[deleted comment]
[deleted comment]
[deleted comment]
[deleted comment]
[deleted comment]
I've heard of these restrictions of GAE, the traffic quota doesn't count. Most people(including me) don't need to download/upload more than 10 GB by day. And since by every gmail account one can apply 10 APPS,while one can have as many as you want gmail accounts,if apjp can allow in its local_properties more than 10 remote servers, the gae traffic quota actually disappears. Despite of all disadvantage of GAE, it also has advantages than other paas:not only will it be always the fastest(because of google's servers distributed in the world),it will also never be blocked. No contry will be so fucked up to block all ips of google. Openshift is a good example to illustrate this, it is now only possible to connect to app of openshift using https in china.I fear that after some time it might be completely blocked by GFW like cloudbees.
Also, why can't apjp connect to gae through ipv6. I'm now using goagent's(you have probably heard about this similar project) ipv6 mood to connect to my gae apps through gogo client's ipv6 in ipv4 NAT traverse tunnel, setting the ipv6 remote server in taiwan, and the speed is at the bandwidth's highest power.
You said that APJP_REMOTE need update for this use, so it sounds like it is technically possible. Actually I(and many others,I believe) would like to pay for this version of apjp, since it will be way much faster than any vpn and offers better anonymity. APJP will then be a more powerful and faster Tor.If it is possible, you can make a big business out of it.
Original comment by Tintinch...@gmail.com
on 12 Mar 2013 at 1:38
1. Speed Problem
The speed depends on your ISP restriction and proxy server location. For me, my
small ISP who buys bandwidth from China Telecom limits to 1/4 of highest speed
about 19:00-01:00 when connecting to foreign websites, especially to U.S. East
Coast. And normally connecting to U.S. West Coast is faster than the East
because of Across-Pacific fiber-optic. Google servers (IP 74.125.*.*) are in
West Coast and Openshif (buys Amazon services) ones are in the East, so to
Google is faster than to Openshift. That's why GAE or HK/JP/SG PaaS and VPS
based proxy is the first choice.
2. IP Changing Problem
Not like Goagent, the respond speed of APJP with multi GAE APP_IDs is slower
than a single one. And GAE will assign every APP_ID connection to one IP (maybe
randomly), that's why you said Google changes IP. I run APJP with just one
APP_ID, then change another one when it's over quota. I have tested Goagent
(without pac file) and Wallproxy (old and new version) with only one APP_ID
too, all run well without IP changing problem. You can test on this site
http://www.hdzone.org where you'll be logout if your IP changes or
http://geoip.flagfox.net.
3. Safety
I don't like Goagent even it's the fastest one and Wallproxy (new version) as
they use pac file or proxy switch browser plugins or directly connect to IP (to
watch Youtue) that I don't know when I'm safe under the proxy. Yet I can change
config to run totally over proxy. But most importantly they don't encrypt data!
So I DO NOT beleive them. I prefer APJP/JAP/Wallproxy-plugins(old version with
password to encrypt data). If you DO care about safety, I suggest you pick up
these three. If you want to say something to fu*k GCD, use JAP + TOR even VPN +
JAP + TOR. Conversely, Goagent is better in the speed and easier to deploy and
use.
4. IPv6
As GFW do not yet have a good way to block IPv6 connection selectively, IPv6
deployment is limited execpt school network (School local network inside China
local network). So proxy with IPv6 supported is better. Hope APJP/JAP to add
this function.
5. Payment
China is a huge market. You can see these years VPN sales is surprising.
However, now GFW can block VPN and SSH selectively through self-studying. On
the other hand, PaaS and VPS based proxies are traffic unique as different
implementations (java/js/python/ruby). I think they are better substitutes. The
only problem is they are hard to deploy for most users. So Jeroen you can think
about this business to sale service to Chinese. You develop program and deploy
to a variety of PaaS and VPS. Users just need an exe file and proxy url and
password, and they can select PaaS or VPS with different price base on
connection speed.
Original comment by westmin...@gmail.com
on 12 Mar 2013 at 8:00
[deleted comment]
[deleted comment]
[deleted comment]
[deleted comment]
[deleted comment]
by the way, I find two paas-paas combination almost without any problem(haven't
tried to download larger file than 200m),
me-Jelastic-openshift-internet,me-jelastic-cloudfoundry-internet.
Original comment by Tintinch...@gmail.com
on 12 Mar 2013 at 11:02
[deleted comment]
[deleted comment]
1. Speed Problem
Goolge's servers that are in the ip range 74.125.*.*,after 18th congress, have
been consistently hassled by GFW, which means that for every 20min Chinese
users cannot access them for a few minutes. Google's servers that perform well
now for chinese are 203.208.46.13*, which are located in China. One reason that
goagent is fast is because it set these ips as default.And china telecom do
have much better performance in connecting to North america servers than china
unicom, but considering the latter's number of users(I think it's the biggest),
the speed of connection to remote servers pose a serious problem for many
chinese.
We cannot get around isp restrictions, but we should try to get the most out of
it. As google had its servers, ip4 and ipv6, in many locations. GAE offers the
best speed experience.The only other way would be to deploy apjp on a chinese
cloud hosting service and use it as proxy for american and european paas,but
due to chinese service qualities, this would absolutely not work.
2. IP Changing Problem
No matter by using goagent or apjp-gae under one appid, tracemyip.org always
shows my ip changes every second. Also, you can test the truth of it by
downloading files from rarefile.com/monsterfile.com and many other share sites;
you can never get the download link by using GAE, because the same ip session
has expired in their 60secs waiting time. Also check a streaming site that
scrutinize your ip constancy:http://vk.com/video166130734_163182395, see if you
can watch this video, the site checks if your ip has changed every minute.
My confusion pops up as I use just-ping.com to see ip of my appspot site, there
it shows the constancy of ip. I guess the ip address I get by ping my GAE and
the Ip address I was distributed by using apjp-gae are not the same. But anyway
I cannot use GAE under one appid to download from file-share sites or streaming
sites that check constancy of ip.
3. Safety
I also have no good impression of PAC file and I never use it. About safety,
there is a thing that bewilders me. Goagent indeed doesn't encrypt data, but as
now all http request to google's ip to look for gae-app are blocked, we only
use https to connect to google, so aren't our data also encrypted? Of course,
APJP is safer, as it encrypts further of this https. Also, I didn't find any
difference in speed between APJP and Goagent under the same google ip.
4. IPv6
I'm using ordinary adsl IPV4 network, but I have installed gogoclient to make
my connection to ipv6 network possible. Gogoclient uses vpn-like method to
transit ipv4 network to ipv6 tunnnel(tunnel broker). By connecting to remote
ipv4-to-ipv6 transition servers in Japan or Taiwan,I have an ipv6 address in
east asia.Next, I set goagent to use ipv6 adresses of google(as apjp cannot
utilise ipv6 of google, even by changing hosts file it is no avail).On speed,
my speed is always what my bandwidth indicates(noticing that always, even
during the traffic jam from 7pm-11pm, it is same, which is a mystery for me).
Under two very different ISPs, this is the same case, always at full speed. And
in this case, my safety is also guaranted by my data traversing one vpn(gogo
ipv6-in-ipv4 tunnel)and then to a ssl connection of google's ipv6 adress.In
fact, when I search in google I would always be asked if I would like to
change to google.tw. The speed isn't compromised by safety,is there a better
solution than this? I suggest you to test this
gogoclient at http://www.gogo6.com/freenet6/tunnelbroker
try one of these four remote servers in East
Asia:60.199.0.228,tb.ipv6.apol.com.tw
tb.ipv6.fetnet.net,203.74.21.89.
You can find perhaps more than a hundred ipv6 address of google here
https://ipv6-hosts.googlecode.com/hg/hosts.
Payment
I agree, come on, Jeroen.
Original comment by Tintinch...@gmail.com
on 12 Mar 2013 at 11:54
[deleted comment]
[deleted comment]
I find an interesting phenomenon of apjp with regard to speed by accident.
I tried to download a 700mb file splitted into 5 parts from rarefiles.net, as
it usually goes with such sharing sites, they regulate download restrictions to
free users. Each IP can download only a part and have to wait 1 hour later to
download another. I was in a café with a 4m adsl wifi of China unicom,
rarefiles.net is blocked by gfw, so I use Jelastic to download in chrome
browser. The speed is 140k/s. I didn't want to wait 1 hour to download another,
so I use firefox for another part, set local ports of apjp from another
jelastic app of the other account of mine differently, and download at the same
time. Now two apjp at the same time were running and surprisingly, two
apjp-jelastic are loading at the same speed, 140k/s,and together
280k/s.However, the system resources are used up in this case, the task manager
shows that I use 96-100% of cpu(my laptop is quite aged, 6 years old), the
biggest monster is the windows kernel service "system".
Two parts were downloaded and my laptop became hot as hell. As my two jelastic
ips cannot download other parts in an hour, I set jelastic as a proxy for
openshift, and use openshift ip to download the third part. It works, the speed
diminishes to 110k/s,but I can bear it. Two apjps were running, but my cpu is
occupied only 2-9% of its resource.
The moral is that tcp connection threads of one apjp is fixed, I wonder if you
can update it to use more threads. This can affect greatly of speed, but only
if it wouldn't use up CPU.
Original comment by Tintinch...@gmail.com
on 12 Mar 2013 at 12:43
To westmin...@gmail.com
These are what hdzone records of my ip when I am using gae-apjp under one
google ip https://203.208.46.131/HTTP and one gae-app
http://www.hdzone.org
IP: 74.125.185.20
上次訪問(last visiting time): 2013.3.11 21:21
現在時間是(now the time is): 21:21
IP: 74.125.185.22
上次訪問(last visiting time): 2013.3.11 21:23
現在時間是(now the time is): 21:23
IP: 74.125.185.23
上次訪問(last visiting time): 2013.3.11 21:28
現在時間是(now the time is): 21:28
http://geoip.flagfox.net.
clearly gae changes its ip every several seconds.I think the reason you are not
forced to logout this forum constantly is that you have preserved its cookie.
http://geoip.flagfox.net shows the same ever changing ip.
About safety again. Generally, it seems that safety and speed were
incompatible, as so many complaints of Tor and I2P show. Datas bouncing through
several server nodes are always slower than direct connection. But due to
Chinese twisted internet enviroment, Paas(faster)+Paas doesn't give me a slower
speed than a single paas. Services seem to be also seriously compromised with
too much sruples of security ,Tor's ruling out of showing flash and java
contents kills most fun of internet. Most normal internet users of China have
never published any sensitive contents on social websites, the most sensitive
contents are at present published on weibo(chinese tweeters) . So most of
Chinese doesn't need so rigorous anonymity and safety like tor or even
vpn-to-ssh-to-tor.As long as one is in china, there can be no strict internet
anonymity if the authority wants to pin you down. We cannot prevent ourselves
from using chinese mobile phone,wechat, qq,internet bank,Alipay.....GFW is a
technologically accomplished project. I believe it can block all cloud services
that apjp may utilize or make the connection excruciatingly slow , as http
request from google to find appspot is already blocked and iranians situations
with regard to tor show. The only reason that it hasn't done its best is that
it knows most chinese didn't do big things out of the wall and block completely
of google will be too scandalous.A tool like apjp at present that requires a
user quite much patience to work will never attract notice of administrators of
GFW, but the reason that cloudbees were blocked might due to its too easy steps
to setup a proxy. As long as one's body is in china,the government always has
means to find who you are, it depends on only whether they have noticed
you.Read this http://www.yunsec.net/a/security/wlgf/2012/0727/10858.html.
Original comment by Tintinch...@gmail.com
on 12 Mar 2013 at 2:31
I do not know about a paying service, I do not think a lot of people (if any)
would pay for APJP.
If people pay for APJP and GFW blocks APJP, what would I do? Repay those
people? I do not live in China so I cannot test GFW, and I cannot help people
in Chinese.
Yes, APJP can be better, if I have enough time and resources (for example: I
cannot make APJP for IOS because I do not have an IPHONE, ..)
Original comment by jeroen.v...@gmail.com
on 13 Mar 2013 at 10:39
[deleted comment]
IP changing
u r right. I "have tested" that's a long time ago. I use other PaaS but GAE, so
don't have that problem.
Business
There are 500+ million internet users in China, and 60%+ use smart phone.
Assume that 1% know your proxy service, and in which 1% will pay for 100¥
(20$) per year, that's 5,000,000¥ (1,000,000$). As the precondition that your
proxy tool is easy enough to use with just an exe file or smart phone apps,
appid and password. The most widely used IM tool is QQ (second Skype ). You can
search more information about GFW via Google or wikipedia.
Original comment by westmin...@gmail.com
on 13 Mar 2013 at 3:38
[deleted comment]
Hi,Jeroen.
Free or not, apjp has changed greatly my experience of internet. I just wish
apjp could be better.
From these two sites you can test if a site is blocked in China.
http://www.websitepulse.com/help/testtools.china-test.html
http://www.greatfirewallofchina.org/
In the attachment is the .pac file of goagent. You can see a non-exhaustive
list of sites blocked by GFW.
I think if GAE can be updated to function as proxy for other paas,two features
need to be further implemented into APJP-GAE for better circumventing gfw.
First- support of IPV6 address.
The reason is that although most people(chinese or not)use ipv4 network,there
are several free tunnel brokers that are available, and it wouldn't be too hard
to set up a tunnel broker server inside china.(and I think it is totally legal).
Second- support of multiple ips.
443 port of ip address of https://google.** (except 203.208.46.13*)will be
blocked by GFW for a few minutes after one connect to it for 20 minutes.It is a
policy that GFW have been using with regard to google for about 15 months. (I'm
not very sure of the accurate date).In this way, Chinese users often have the
wrong impression that google's service is not good enough. So goagent uses
multiple ips to connect to google. In goagent's console, it shows that it
connects to 4 ips at the same time and if there's no response from these 4 ips,
it changes to the next 4 ips until there is response. Goagent also can do it
for ipv6 address.
The second file in the attachment is my goagent's configuration file, you can
see that it supports multiple ips of ipv4 and ip6.
I think a gae-based proxy, apjp or others , can be more intelligent and easy to
use for Chinese by adding an additional little software. I mean a software that
includes an exhaustive ip list of google at present, both ipv4 and ipv6, it can
test their ping time, produce a ranking of from their ping time and add these
fast google ips into the apjp_local.property.
Though I'm not sure if by using different row of multiple ips to connect to
google then to my-appspot site one can download a large file.
Anyway, apjp has done a great job in combining both speed and safety as a
proxy. I just suggest several features that it can update that will better fit
Chinese internet environment.(Maybe also other contries that use Chinese GFW
based firewalls,such as Iran).
Original comment by Tintinch...@gmail.com
on 13 Mar 2013 at 9:13
Attachments:
PS:The ip range from 203.208.46.0 to 203.208.46.255 all belong to google and
are located in Beijing. It will leave too much of a speed gap to not set up gae
as proxy to other paas. Multiple support of Ipv4 addresses also will be
wonderful.
Original comment by Tintinch...@gmail.com
on 14 Mar 2013 at 1:13
I have updated APJP_REMOTE_JAVA_APPENGINE (see
https://www.dropbox.com/sh/zltcyty6sdidwbl/8Xsa4UiWaB).
In APJP_REMOTE.properties you configure:
- APJP_REMOTE_HTTP_SERVER_REQUEST_URL == http://../HTTP
- APJP_REMOTE_HTTPS_SERVER_REQUEST_URL == http://../HTTPS
In APJP_LOCAL.properties you configure:
- APJP_REMOTE_HTTP_SERVER_X_REQUEST_URL ==
http://application-id.appspot.com/HTTP2HTTP
- APJP_REMOTE_HTTPS_SERVER_X_REQUEST_URL ==
http://application-id.appspot.com/HTTPS2HTTPS
Now APJP_LOCAL is configured to send requests to http://../HTTP via
http://application-id.appspot.com/HTTP2HTTP (and http://../HTTPS via
http://application-id.appspot.com/HTTPS2HTTPS)
Can you test?
Original comment by jeroen.v...@gmail.com
on 14 Mar 2013 at 10:31
[deleted comment]
Great. I'm excited to deploy the new update.Haven't had enough time to test all
paas, the following is what I find.
As china has blocked direct connection to
http://application-id.appspot.com/HTTP2HTTP, I use
APJP_REMOTE_HTTP_SERVER_X_REQUEST_URL = https://a-google-china-located-ip
/HTTP2HTTP, key=Host, Value=my-appid.appspot.com. I also compare this
configuration's effect with connecting to APJP_REMOTE_HTTP_SERVER_X_REQUEST_URL
= http://application-id.appspot.com/HTTP2HTTP through a ssh-based http, seems
there's no difference.
What a single gae or paas can do, this gae-paas can also do it, I have tested
youtube,vimeo, download a long high-definition video from keepvid.com, there's
no problem.
My ip keeps constancy.
With regard to downloading from file-sharing sites that single paas used to be
able to do, I haven't succeeded with gae-paas.
I tried these three files: http://uploaded.net/file/5onqhwkp
http://depositfiles.com/files/rf7ll4dsm
http://ryushare.com/bea14da7467/InReaSe76.part1.rar.
I have attached report in apjp console when it fails to download. I have tried
several other paas on downloading these files, didn't have time to take
snapshot of their reports, but https://goolgeip/http2http return 500 and
request/method=" "all shows.
I can use the updated apjp to watch videos on vk.com, I couldn't do it with
using gae or gae-to-paas before.But fails to watch videos on purevid.com and
mixturecloud.com. Check purevid.com/v/19159w1yzomnw345083/. Have also attach a
report a jelastic failure to watch purevid in the attachment.
despite of these, great job.I never expect an update so soon.
Original comment by Tintinch...@gmail.com
on 15 Mar 2013 at 1:30
Attachments:
Download from http://uptobox.com/dd3h8rqk24uh:
With GAE I can never get around the captcha code check, each time I type the
right code it directs me to a new session of 60 seconds waiting to type again
capcha code.
With the update http2http-Heroku I can get to the download link, but the
downloading grinds to a halt at 32mb, the size of the restriction of GAE
already mentioned. And at its download failure moment also pops up the "500 for
url:https://googleip/HTTP2HTTP" for five times.
I report this example because it is the first time I find a gae based proxy can
download from a filesharing site. Though the download stops at 32mb.
Original comment by Tintinch...@gmail.com
on 15 Mar 2013 at 5:25
Attachments:
This is a problem I cannot correct :(.
When you download a file from download sites like uploaded.net, you can only
download the file once. When the file is >32MB, APJP_REMOTE_JAVA_APPENGINE
returns a status 500, and APJP_LOCAL_JAVA will then download the file with a
lot of downloads who are <32MB. But download sites like uploaded.net do not
like that :(.
I think a good PAC will allow you to complete CAPTCHAs like
http://www.google.com/recaptcha (if they are not blocked by GFW).
The restrictions of APPENGINE are too big, and the advantages of HTTP2HTTP and
HTTPS2HTTPS too little, so the update will not be added to APJP.
Also configuring HTTP2HTTP and HTTPS2HTTPS would be too difficult for a lot of
people.
Original comment by jeroen.v...@gmail.com
on 15 Mar 2013 at 7:29
I see, is it that filesharing sites require their file to be downloaded once
for all and gae cannot spit it? Anyway, gae seems to have too many restrictions.
what about streaming site like purevid, it seems that the requested video url
is quite different from sites say, youtube, there is also no way to get around
this?
The capcha doesn't pose a problem for http2http.
I think configuration of HTTP(S)2HTTP(S) will not be difficult as long as there
is good wiki instructions. But at present the advantage is too little.
Anyway, thanks for the (unofficial) update and your replies.
Original comment by Tintinch...@gmail.com
on 15 Mar 2013 at 7:57
Sorry, I don't understand software programming at all, but maybe blobstoreapi
can make gae deal with large file?
https://developers.google.com/appengine/docs/java/blobstore/overview
Also, is there someway to slice the requested file into multiple chunks for
downloading, and then reconstructing them into one?
Original comment by Tintinch...@gmail.com
on 15 Mar 2013 at 8:50
The download site creates a session when you start downloading a file (after X
seconds). When you stop downloading that file (for whatever reason), the
download site destroys that session.
Without that session you cannot download.
Other sites also use sessions, but how, where and when is site dependent.
The restrictions are with the URL Fetch API, which downloads the files. The
Blobstore API cannot help.
You can only use the APPENGINE API to download files, so you cannot reuse
connections to download files :(.
Original comment by jeroen.v...@gmail.com
on 15 Mar 2013 at 9:39
:〔 I think openshift and cloudfoundry have much less restrictions than GAE.
So is there a way to make, when download a file or watch a video...that
requires a single session, the url fetch response to be not immediate
transmitted to paas-gae-me, but first fetched and kept in paas, then gae asks
the paas to load it? Though this reminds me a leech site. Or just let the
paas,say openshift, fetch and keep and give a code to every 32mb data from a
single session and delete this 32mb after a certain time, by this code the gae
can fetch it and know to combine it with other 32mb files loaded from openshift.
Also, if i set up ten http2http gae apps all with the same remote server,say an
openshift app, when openshift fetch a url, for example,230mb,can openshift be
instructed to forward this request every gae-app to fetch 1/10 part of it
simultaneously?
Original comment by Tintinch...@gmail.com
on 15 Mar 2013 at 3:37
No, that is not possible, the other PAAS also have restrictions (but less than
APPENGINE).
Original comment by jeroen.v...@gmail.com
on 16 Mar 2013 at 10:01
Original issue reported on code.google.com by
Tintinch...@gmail.com
on 10 Mar 2013 at 2:10