karlcheong commented 11 years ago

記得很久前有人就問過在路由器如何設置Unblock Youku 後來更有一篇教程, 通過squid來達成Unblock Youku功能

對於一般家居用的路由器來說,都是基於MIPS LE(Boardcom)或MIPS BE(Atheros) unblock youku伺服器使用之Node.js對MIPS支援性很差, 很難用來運行轉發代理而 squid 倒是可以在家用路由器運行, 不過一來squid設定複雜我不會設, 二來我正好在 #80 中為了給Android也能解鎖, 用privoxy 運行轉發代理可以直接把設定搬過來用, 所以這個教程使用的軟件為privoxy

要求: 已安裝OpenWRT系統的路由器

1 SSH進去路由, 然後輸入指令安裝 privoxy opkg update opkg install privoxy

2 把規則文件及basicsetting設定文件, 放到 /etc/privoxy

https://github.com/karlcheong/unblock_youku_privoxy_rules

3 輸入指令啟動 privoxy /etc/init.d/privoxy start

(可選:)輸入指令, 使路由啟動時自動運行 privoxy 代理 /etc/init.d/privoxy enable

4 把要解鎖的裝置設定代理為路由區域網IP:8123

karlcheong commented 10 years ago

@shine12 路由系統上的resolv.conf 不能使用 192.168.11.1 / 127.0.0.1 (就是你自己架的, 修改過紀錄的DNSMASQ) 不然這樣privoxy就會不斷連接自己, 陷入死循環了

/cifs2/optware/resolv.conf 這個應該是Dnsmasq用的resolv.conf, 不是系統使用的DNS設定

shine12 commented 10 years ago

etc資料夾下 dnsmasq.conf中有resolv-file=/etc/resolv.dnsmasq 並且這resolv.dnsmasq裡面是nameserver 192.168.1.1>>>這是dnsmasq用的resolv.conf resolv.conf 中是nameserver 127.0.0.1>>>這是系統上的resolv.conf 就是這樣看嗎?

我的privoxy 監聽192.168.11.1:80在結果上 resolv.dnsmasq resolv.conf 應該改成怎樣呢?

karlcheong commented 10 years ago

系統resolv.conf里的DNS伺服器改為你ISP的DNS 不能用本機的DNS

dnsmasq的resolv 文件也不需要nameserver 192.168.1.1

expkids commented 10 years ago

Karlcheong兄, 依據您提點的步驟,剛架好 privoxy + dnsmasq router 在openwrt內, 進行反向代理. 經測試, 一切正常...可惜, 在router重新啟動後.."反向代理" 便即失效. 這是否正常, 又或過程中有遺漏. 奇怪的是: 當我用Putty ssh 進入router..把privoxy以--no-daemon運行"privoxy, --no-daemon /etc/privoxy/config". 運行當中並無error出現. 之後又可反向代理,一切正常.

whuhacker commented 10 years ago

@BryanHK 不要用 --no-daemon

啟動 privoxy /etc/init.d/privoxy start

路由啟動時自動運行 privoxy 代理 /etc/init.d/privoxy enable

expkids commented 10 years ago

謝謝回答, 今晚斷電再試, 但為何router在重新啟動後.."反向代理" 便告失效.

whuhacker commented 10 years ago

@BryanHK 因為 privoxy 服務在開機後沒有自動運行，你需要使用上面的指令將其設置為開機自動啟動

expkids commented 10 years ago

試過:輸入指令啟動 privoxy /etc/init.d/privoxy start /etc/init.d/privoxy enable No Luck... 再試: 重啟Dnsmasq /etc/init.d/dnsmasq reload Again no luck Try privoxy --no-daemon /etc/privoxy/config again: Ok, proxy run ..><..

whuhacker commented 10 years ago

@BryanHK /var/log is on tmpfs and cleans on reboots, so privoxy can't create folder /var/log/privoxy and fails. Just change logdir in /etc/privoxy/config to /var/log and finally it starts.

expkids commented 10 years ago

Problem fixed . Thanks whuhacker and karlcheong ..^^..

simonleungs commented 10 years ago

有沒有人懂怎用openwrt 中 release / renew ip 嗎? 雖然離題了,但真心求教T.T

karlcheong commented 10 years ago

@simonleungs 你在做什么嗎?

simonleungs commented 10 years ago

我剛刷了openwrt, 但我在用pccw ,網上說某字頭的ip比較快,所以想試試

simonleungs commented 10 years ago

@karlcheong 請問可以怎做??十萬個感激!!!

yifengzhou commented 10 years ago

想问个问题。我没有用家里总的路由器，怕影响速度。我弄了一台我如同wrt54gl装了openwrt, 任何连到路由器的LAN口，按下面的步骤能工作吗？

1 安裝 privoxy 2 把規則文件及basicsetting設定文件, 放到 /etc/privoxy 3 在openwrt界面上设路由器啟動時自動運行 privoxy 4 把要解鎖的裝置設定代理為路由區域網IP:8123 谢谢

kukat commented 10 years ago

@shine12 请问在 tomtato 上面成功了吗？还请分享

kukat commented 10 years ago

@karlcheong Asus RT-N56U + 3rd party firmware + Entware 已安装 privoxy 3.0.21 和 basicsetting 经测试直接使用 web proxy 192.168.1.1:8123 是可以的。但 dnsmasq 不起作用，firmware 里的 dnsmasq 没有 init 文件。不知道怎么实现你提到的第4步。

/home/root # find / -name 'dnsmasq*'
/etc/dnsmasq.conf
/etc/dnsmasq.conf.default
/etc/storage/dnsmasq
/etc/storage/dnsmasq/dnsmasq.conf
/tmp/dnsmasq.leases
/usr/sbin/dnsmasq
/var/run/dnsmasq.pid

/home/root # find / -name 'resolv*'
/etc/resolv.conf
/var/lock/resolv.lock

/home/root # find / -name 'privoxy*'
/opt/var/log/privoxy
/opt/etc/privoxy
/opt/lib/opkg/info/privoxy.control
/opt/lib/opkg/info/privoxy.conffiles
/opt/lib/opkg/info/privoxy.list
/opt/sbin/privoxy
/var/log/privoxy

bebeboy commented 10 years ago

Appreciate your great afford. I have setup my own privoxy and able to watch the videos with my router G300NH. But I found that the pre-download function is not working either from PC or mobile app. Is that something I have to check with? Thanks in advance.

karlcheong commented 10 years ago

@kukat 你先看看你路由系統的resolv.conf 看看是不是用了本身路由上的dnsmasq (例如 192.168.1.1) 作DNS解析是的話就改掉 (改為ISP 的DNS或GOOGLE DNS或OPENDNS)

我上面的方法單純只給同版本openwrt用的, 不是相同的系統就得變通一下思路我都寫在這里, 還有 #112 里了

karlcheong commented 10 years ago

@bebeboy Is pre-download means 緩存(downloading/saving video on the disk) ?

Not all videos can be 緩存 in the pc/mobile apps the video service providers might restrict the right to download certain videos on their site.

Have a look on my privoxy filter files, it contains some rules for bypassing the download restriction in some mobile apps (by changing the download restriction flag in the API server responses), but some domain names required by the rules is not in the basehosts list by default and also they are not guarantee to work. (it doesn't work on youku apps because of API server restriction)

If you're on PC, you can easily download the target video by using some video downloading site. (eg flvxz.com)

If you still want to bypass the restriction on specific mobile apps, Please lists the names of videos and the name/platform of client which you have problem with. I will answer at my best. (eg: 来自星星的你 / tudou android phone)

bebeboy commented 10 years ago

Thanks karlcheong. I think you have told me the story, exactly I am not able to download the videos from youku apps. One more questiosn, it's there anyway I can skip youku video advertisement?

karlcheong commented 10 years ago

You may refer to https://github.com/zhuzhuor/Unblock-Youku/issues/112#issuecomment-42157758 the youku apps m3u8 server is pl.youku.com

to make it simple, add pl.youku.com item into basehosts and trustlist and you are good to go.

bebeboy commented 10 years ago

Thanks again.

Will give it a try. Cheers.

bebeboy commented 10 years ago

It's not working suddenly since last day night with the following error.

抱歉，連接失敗　 Ref 2003

Did someone encountered the same?

karlcheong commented 10 years ago

@bebeboy try to update the rules files to the latest and test again Also please stat the apps/website you encountered the problem with

tywtyw2002 commented 10 years ago

楼主，你这个dns劫持的方法，想当是反向代理，但是你有考虑过如果在js层面有一些非标准端口或者websocket的操作呢。

karlcheong commented 10 years ago

@tywtyw2002 目前來說, 在做反向代理過程中, 沒看到非标准端口/websocket 非标准端口我想可能性不高.. 因為不少用戶都是在防火牆後的...只能跑80/443 websocket 等國內的XP/IE6~9用戶都換成IE 10再算吧...

sequoiar commented 10 years ago

@karlcheong 页面里的跨域链接咋处理

karlcheong commented 10 years ago

页面里的跨域链接..那跟這個反向代理也沒有關係吧?

bebeboy commented 10 years ago

QQ Music have "no such file or directory error". Something changed ?

karlcheong commented 10 years ago

@bebeboy 無看到有什么改變呢? 能詳細點說明嗎?有截圖就更好了.

bebeboy commented 10 years ago

Thanks for your reply. It may be QQ music issue and I cannot simulate the error today. WIll let you know if it's happen again.

Besides, I have a question about the privoxy config file. Your config file has commented out the below statement, but by default, most of those should be on. Any means for you to turn it off or any contradict with your filter/action rules if I open it ? Thanks a lot.

actionsfile match-all.action # Actions that are applied to all sites and maybe overruled later on.

actionsfile default.action # Main actions file

actionsfile user.action # User customizations

trustfile trust

karlcheong commented 10 years ago

@bebeboy

trustfile is not on by default. It sets a url whitelist for the proxy, only urls that match on the list can pass through the proxy. Those three actionfiles is part of the sample config files bundled with the privoxy software. They have included some sample action rules, but no urls is specified in the rules therefore actually it is not functional by default.

It is OK to enable it and add urls to the included rules. The privoxy config sort the action rules by order. If one url matches multiple contradicted rules, (eg forward-override) the rules at the bottom should be used

bebeboy commented 10 years ago

Thanks to clarify.

I am interest to develop my own filter / actions, may I know any tool can ease the development ? Or should you just base on the debug information ?

karlcheong commented 10 years ago

Well, usually I use Fiddler debug proxy (use privoxy as upstream of Fiddler) to check the urls and its contents, modify privoxy actions rules and check the result on the fly. (Privoxy takes the change in actions and filter files immediately, no restart needed)

I don't look the debug log in the privoxy usually.

To build your own privoxy action/filter files:

Read the manual carefully, especially the actionsfileand filter part
Know some basic regular expression syntax.
Make a good use of look up action apply to urls tool in the privoxy control panel
There are different debug log option in the config, you can enable based on you need.

lamplamp commented 10 years ago

类似Kucat, 使用 DD-WRT + Optware, 安装 privoxy 和 basicsetting. Browsers使用 Forward proxy 192.168.1.1:8123 是可以的。但是如何设置PBO的媒体播放器dvdplayer使用代理? export http_proxy=http://192.168.1.1:8123 不行。HTTP_PROXY 也不行。
可以设置privoxy 使用 reverseproxysetting. DD-WRT也有 dnsmasq ，但怎么实现第4步? 本人菜鸟，不知楼主或用过 PBO / DD-WRT dnsmasq 的大侠能否指点迷津?

karlcheong commented 10 years ago

1 第一次看到的盒子呢..可能是該程式有自己的方法設定proxy (在設定檔里?), 又或者根本不支持proxy? 可以試試ddwrt下使用iptables 把PBO的http流量都強制導入到privoxy

2 4步是因為 dnsmasq 啟動時, 它的init script會修改 resolv.conf 里的dns 設定到 127.0.0.1 (就是dnsmasq自己). 要是知道你需要使用的DNS的地址, 也可以這樣做: 在 wan up script 加入以下

echo nameserver 8.8.8.8 > /tmp/resolv.conf
echo nameserver 8.8.4.4 >> /tmp/resolv.conf
echo nameserver 4.2.2.1 >> /tmp/resolv.conf
echo nameserver 4.2.2.2 >> /tmp/resolv.conf

wan up script 看這里 /tmp/resolv.conf 是 dd-wrt resolv.conf的實際位置 8.8.8.8等是想使用的WAN DNS (只需指定一個DNS的話可以去掉其他三行)

lamplamp commented 10 years ago

多谢指点. 我会试一下.

bebeboy commented 10 years ago

http://sports.cntv.cn/live/cctv5/ is not working anymore. Does it correct?

bebeboy commented 10 years ago

Hi karlcheong.

Another question about filter setup. I have tried to filter / skip a div in html with the a filter rule But this not working, Can you show me some hints ? Thanks in advance.

untitled

karlcheong commented 10 years ago

@bebeboy better to have the link of the page..

bebeboy commented 10 years ago

Let say http://www.yyets.com/ and I want to remove the left advertisement.

I added a filter rule below but it's not working.
untitled

karlcheong commented 10 years ago

@bebeboy for removing ads on yyets homepage

actionfile:

{ +filter{yyetsads} +force-text-mode +forward-override{forward .} }
.yyets.com
{ +block{Reason: baiduads} }
#for removing the baidu ads on yyets
res.yyets.com/ads/.*
cpro.baidustatic.com/cpro/ui/.*\.js.*
pos.baidu.com/

filterfile:

FILTER: yyetsads for yyets
s|<div id="floatAD_.*</div>||Usg
s|<div class="games-rec-list">|<div class="games-rec-list" style="display:none;">|g
s|\.ads_partner_list(.*);|.ads_partner_list$1; display:none;|Ug

bebeboy commented 10 years ago

多谢指教。

leira commented 10 years ago

@kukat I use RT-N56U too, until reading the source from https://code.google.com/p/rt-n56u/, I found that the logic adding “nameserver 127.0.0.1” into /etc/resolv.conf was hardcoded in the binary code of /sbin/rc, there is no way to avoid that. The way I solved it, was to rebuild the /etc/resolv.conf file in /etc/storage/started_script.sh, but it also means you lost the capability to change the DNS servers from web GUI.

Add the following lines into /etc/storage/started_script.sh:

echo nameserver 8.8.8.8 > /tmp/resolv.conf echo nameserver 8.8.4.4 >> /tmp/resolv.conf

save the change by calling:

mtd_storage.sh

reboot to apply the change.

LogicoZone commented 10 years ago

@karlcheong

I've tried configured everything with reverse proxy settings but still doesn't work. I also tried "http://ipservice.163.com/isFromMainland" to verify on my computer and the return is false. How can I verify if my settings is working? Or at least, to know if it goes through privoxy properly?

I think my question is that all devices do not need configuration? if I don't configure the browser to use the privoxy's port, it doesn't work. How can I make it work w/o configuration change on devices? I'm kind of confused.

More detail. I checked the dnsmasq and it does load the basehosts.

root@OpenWrt:/tmp# dnsmasq -C /var/etc/dnsmasq.conf -d dnsmasq: started, version 2.71 cachesize 150 dnsmasq: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-auth no-DNSSEC dnsmasq-dhcp: DHCP, IP range 10.0.1.100 -- 10.0.1.249, lease time 12h dnsmasq: using local addresses only for domain lan dnsmasq: reading /tmp/resolv.conf.auto dnsmasq: using local addresses only for domain lan dnsmasq: using nameserver 75.75.75.75#53 dnsmasq: using nameserver 75.75.76.76#53 dnsmasq: using nameserver 2001:558:feed::1#53 dnsmasq: using nameserver 2001:558:feed::2#53 dnsmasq: read /etc/hosts - 1 addresses dnsmasq: read /tmp/hosts/dhcp - 0 addresses dnsmasq: read /tmp/hosts/odhcpd - 0 addresses dnsmasq: read /etc/privoxy/basehosts - 93 addresses dnsmasq-dhcp: read /etc/ethers - 0 addresses

Then I checked the dns resolving from my computer and it does resolve fine to my router local net ip.

Leo-rMBP:~ sliu$ dig ipservice.163.com

; <<>> DiG 9.8.3-P1 <<>> ipservice.163.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32656 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION: ;ipservice.163.com. IN A

;; ANSWER SECTION: ipservice.163.com. 0 IN A 10.0.1.202

;; Query time: 25 msec ;; SERVER: 10.0.1.202#53(10.0.1.202) ;; WHEN: Sun Sep 14 17:05:57 2014 ;; MSG SIZE rcvd: 51

However, I don't see any activity in the privoxy log. It seems it does nothing and the privoxy does listen on port 80 on router. Somehow, request to the privoxy seems not doing anything. Any hints on the problem?

Thanks.

akpotter commented 9 years ago

感謝樓主的文章跟 rule, 昨天在家裡的 openwrt 測試 reverse proxy 成功了。利用這個方法，小米盒子連上 openwrt router 也可以不需要改任何設定，收看 youku 等源的影片。在安裝的時候遇到一些問題拋磚引玉一下:

預設安裝新版 opkg install privoxy 以後，用 /etc/init.d/privoxy start 開看起來會抓到 uci 的 config 而不是 /etc/privoxy/config，後來發現問題以後手動執行 privoxy /etc/privoxy/config 就成功了。
想要在開機帶起來的話也可以將 "privoxy /etc/privoxy/config" 加入 /etc/rc.local 的 exit 前面行即可。

bba56k commented 9 years ago

ddwrt可以吗？能post一下你的详细步骤吗？多谢

On Thursday, January 8, 2015, akpotter notifications@github.com wrote:

感謝樓主的文章跟 rule, 昨天在家裡的 openwrt 測試 reverse proxy 成功了。利用這個方法，小米盒子連上 openwrt router 也可以不需要改任何設定，收看 youku 等源的影片。在安裝的時候遇到一些問題拋磚引玉一下:

1.

預設安裝新版 opkg install privoxy 以後，用 /etc/init.d/privoxy start 開看起來會抓到 uci 的 config 而不是 /etc/privoxy/config，後來發現問題以後手動執行 privoxy /etc/privoxy/config 就成功了。 2.

想要在開機帶起來的話也可以將 "privoxy /etc/privoxy/config" 加入 /etc/rc.local 的 exit 前面行即可。

— Reply to this email directly or view it on GitHub https://github.com/zhuzhuor/Unblock-Youku/issues/106#issuecomment-69147669 .

karlcheong commented 9 years ago

@bba56k 如果是只改dnsmasq 把特定域名指向我們的代理的話還好，但想要架設privoxy／其他HTTP代理的話， ddwrt 的方法會跟openwrt有相當不同。。

相比openwrt，因為ddwrt的系統分區是唯讀的，所以軟件包和設定只能放在路由內置jffs2分區／USB／samba上。（路由內建jffs2分區空間和USB接口在便宜的路由幾乎沒有，因為系統分區已佔用了大部分空間，而用samba的話還不如把privoxy放到跑samba的機器上運行）還有DDWRT是沒有預裝opkg管理器的，得手動安裝，比較麻煩也易出問題。

簡單來說，要用ddwrt弄privoxy的話，假設你的路由有足夠內置jffs2空間/USB記憶棒，我會這樣做： 1 安裝opkg，privoxy 和他們的依賴包以及privoxy的rules到jffs2分區，把basehosts也放到jffs2 2 ddwrt中的dnsmasq設定中加入上面的addn-hosts, 還有下面額外設定使它不監聽 lo 本機介面

interface=<你的LAN區域網絡介面編號,一般是br0>
except-interface=lo

3 在boot script 再運行一個標準設定的 dnsmasq ，只監聽 lo 本機介面

bba56k commented 9 years ago

你的openwrt路由器已经自动代理了？任何连接的客户端都自动走unblock youku? 盒子上，什么也不用设置？

能把你参考的文章链接或者＠我一下吗？我找个openwrt路由试试！

多谢 On Jan 9, 2015 10:49 AM, "karlcheong" notifications@github.com wrote:

如果是只改dnsmasq 把特定域名指向我們的代理的話還好，但想要架設privoxy／其他HTTP代理的話， ddwrt 的方法會跟openwrt有相當不同。。

相比openwrt，因為ddwrt的系統分區是唯讀的，所以軟件包和設定只能放在路由內置jffs2分區／USB／samba上。

（路由內建jffs2分區空間和USB接口在便宜的路由幾乎沒有，因為系統分區已佔用了大部分空間，而用samba的話還不如把privoxy放到跑samba的機器上運行）還有DDWRT是沒有預裝opkg管理器的，得手動安裝，比較麻煩也易出問題。

簡單來說，要用ddwrt弄privoxy的話，假設你的路由有足夠內置jffs2空間/USB記憶棒，我會這樣做： 1 安裝opkg，privoxy 和他們的依賴包以及privoxy的rules到jffs2分區，把dnsmasq的設定也放到jffs2 2 ddwrt中的dnsmasq設定中加入上面的addn-hosts, 還有下面額外設定使它不監聽 lo 本機介面

interface=<你的LAN區域網絡介面編號,一般是br0> except-interface=lo

3 在boot script 再運行一個標準設定的 dnsmasq ，只監聽 lo 本機介面

— Reply to this email directly or view it on GitHub https://github.com/zhuzhuor/Unblock-Youku/issues/106#issuecomment-69370644 .

uku / Unblock-Youku

打造簡便unblock youku路由器(使用openwrt + privoxy) #106

actionsfile match-all.action # Actions that are applied to all sites and maybe overruled later on.

actionsfile default.action # Main actions file

actionsfile user.action # User customizations

trustfile trust