Open nexusofdoom opened 5 years ago
Hi @nexusofdoom
Sorry. I don't understand your question. Please explain in detail.
Thanks.
caching nginx box setup like so.. online host uses Prerequisites for Token Authentication from akimia it addess this to the request acl=/*~hmac=6a10b3f602ffde88c02cd1b89665bfdfdc0fc84c3cf7049752931ad732654fab
the client points to my nginx caching box and ask to get the file but it never gets a hit on the key.
is there a way to rewrite the new token to the end of the request ? or something more I can do?
this is the url it requests "akamai-gamecdn.blackdesertonline.com/live001/game/download/131/BlackDesertOnline885.bcab?begin=1238753107&end=1247683342&hdnts=st=1537229206exp=1537230106acl=/live001/game/download/!/live001/game/patch/~hmac=c2d249db5aff5d6940e03447517ba12b41fdb9aa204cd3c762a9265f445beade"
and below is what the key shows that it miss /live001/game/download/131/BlackDesertOnline885.bcab
Hello
What is the response code for this request?
"akamai-gamecdn.blackdesertonline.com/live001/game/download/131/BlackDesertOnline885.bcab?begin=1238753107&end=1247683342&hdnts=st=1537229206exp=1537230106acl=/live001/game/download/!/live001/game/patch/~hmac=c2d249db5aff5d6940e03447517ba12b41fdb9aa204cd3c762a9265f445beade"
where do you want me to get the response code ? I have httpry installed along with I can look at the nginx log or do you want me to run wireshark on the client pc?
note in httpry its code 200
You can get it using curl. I want to determine if the request is returned 200.
Ok will do when I get back from work tonight. thanks.
Sorry, I can only judge by my experience. I think you should request the origin website, check if the response is allowed caching. And also send me a copy.
direct test talking to there server curl "akamai-gamecdn.blackdesertonline.com/live001/game/download/131/BlackDesertOnline885.bcab?begin=1238753107&end=1247683342&hdnts=st=1537229206~exp=1537230106~acl=/live001/game/download/!/live001/game/patch/~hmac= c2d249db5aff5d6940e03447517ba12b41fdb9aa204cd3c762a9265f445beade" curl : An error occurred while processing your request. Reference #199.5ef00a17.1537399717.1bba468b At line:1 char:1
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
if I use lancache I get this, note it saves files to the cache and the client gets the file but the next time it downloads the acl= changes and it comes back as a miss in nginx and it re-downloads the file all over again.
curl "akamai-gamecdn.blackdesertonline.com/live001/game/download/131/BlackDesertOnline885.bcab?begin=1238753107&end=1247683342&hdnts=st=1537229206~exp=1537230106~acl=/live001/game/download/!/live001/game/patch/~hmac= c2d249db5aff5d6940e03447517ba12b41fdb9aa204cd3c762a9265f445beade"
StatusCode : 200 StatusDescription : OK Content : À(Y Ð Ü·0 4 À" 3 òR ô7 x/ Â aq lg @ç }y ´ÞLÏ ¬N P N º@ÔÂ üO P N ùm6 LQ pC 0 3ë+© ¼ = x ¨Aß TÒ C ... RawContent : HTTP/1.1 200 OK Connection: keep-alive X-Upstream-Cache-Status: HIT Accept-Ranges: bytes Content-Length: 8930236 Content-Type: text/plain Date: Wed, 19 Sep 2018 23:25:32 GMT ETag: "5518bf566eb7... Forms : {} Headers : {[Connection, keep-alive], [X-Upstream-Cache-Status, HIT], [Accept-Ranges, bytes], [Content-Length, 8930236]...} Images : {} InputFields : {} Links : {} ParsedHtml : mshtml.HTMLDocumentClass RawContentLength : 8930236
Hello
I noticed $request_url is used as key in your nginx conf. $request_uri : full original request URI (with arguments)
this is the testing config I am playing with. /etc/nginx/vhosts-enabled/lancache-pearlabyss.conf
server { listen lancache-pearlabyss deferred default; servername pearlabyss ;
access_log /srv/lancache/logs/Access/pearlabyss.log main buffer=128k flush=1m;
access_log /srv/lancache/logs/Keys/pearlabyss.log keys_uri buffer=128k flush=1m;
error_log /srv/lancache/logs/Errors/pearlabyss.log;
# Default Node
include lancache/resolver;
#include lancache/cache-key-range;
location / {
set $no_cache 0;
if ($request_uri ~* "(/live001/game/config/config.patch.version)") {
set $no_cache 1;
}
proxy_no_cache $no_cache;
proxy_cache_bypass $no_cache;
proxy_bind 10.0.4.20;
proxy_ignore_headers Expires Cache-Control;
#proxy_set_header Range $http_range;
#proxy_set_header If-Range $http_if_range;
#proxy_cache_key "$server_name$uri";
#proxy_cache_key "$scheme$request_method$host$uri$is_args$args";
#proxy_cache_key "$uri$is_args$args$slice_range";
#proxy_set_header X-Real-IP $remote_addr;
#proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#proxy_cache_valid 200 206 320d;
#proxy_cache_key $scheme$host$proxy_host$request_uri;
#proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#proxy_set_header Host $host;
#proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 150;
# Use Origin cache
proxy_cache pearlabyss;
include lancache/proxy-cache;
#include lancache/cache-range;
include lancache/cache-key-default;
}
}
Your should use $uri instead of $request_uri as proxy_cache_key.
so change this include lancache/cache-key-default; } }
to
include lancache/cache-key-uri;
}
}
I did this
server { listen lancache-pearlabyss deferred default; servername pearlabyss ;
access_log /srv/lancache/logs/Access/pearlabyss.log main buffer=128k flush=1m;
access_log /srv/lancache/logs/Keys/pearlabyss.log keys_uri buffer=128k flush=1m;
error_log /srv/lancache/logs/Errors/pearlabyss.log;
# Default Node
include lancache/resolver;
#include lancache/cache-key-range;
location / {
set $no_cache 0;
if ($request_uri ~* "(/live001/game/config/config.patch.version)") {
set $no_cache 1;
}
proxy_no_cache $no_cache;
proxy_cache_bypass $no_cache;
proxy_bind 10.0.4.20;
proxy_ignore_headers Expires Cache-Control;
#proxy_set_header Range $http_range;
#proxy_set_header If-Range $http_if_range;
proxy_cache_key "$uri";
once i did that this is the error I get in the client
httpry on the nginx server 2018-09-19 21:02:01 10.0.4.45 10.0.4.39 > GET akamai-gamecdn.blackdesertonline.com /live001/game/download/132/BlackDesertOnline891.bcab?begin=0&end=12&hdnts=st=1537408917~exp=1537409817~acl=/live001/game/download/!/live001/game/patch/~hmac=a1280702ad1721200608c2af28d41b3156e7c46208f4dc5c43f627e6cd5b773b HTTP/1.1 - - 2018-09-19 21:02:01 10.0.4.39 10.0.4.45 < - - - HTTP/1.1 200 OK 2018-09-19 21:02:01 10.0.4.45 10.0.4.39 > GET akamai-gamecdn.blackdesertonline.com /live001/game/download/132/BlackDesertOnline891.bcab?begin=13&end=359540&hdnts=st=1537408917~exp=1537409817~acl=/live001/game/download/!/live001/game/patch/~hmac=a1280702ad1721200608c2af28d41b3156e7c46208f4dc5c43f627e6cd5b773b HTTP/1.1 - - 2018-09-19 21:02:01 10.0.4.39 10.0.4.45 < - - - HTTP/1.1 200 OK 2
if i change the hmac= from 773b to 733a i get this message in chrome An error occurred while processing your request. Reference #199.5ef00a17.1537409068.1bff07c8
curl "akamai-gamecdn.blackdesertonline.com/live001/game/download/131/BlackDesertOnline885.bcab" curl : An error occurred while processing your request. Reference #199.3cb2ef50.1537409184.76a1387 At line:1 char:1
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebExc
eption
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
curl -v "akamai-gamecdn.blackdesertonline.com/live001/game/download/131/BlackDesertOnline885.bcab"
It will return 403. It means that the certification did not pass.
I think you should be aware of what kind of request your application make to the nginx proxy, including the headers of each request. Then manually curl these request.
curl -v "akamai-gamecdn.blackdesertonline.com/live001/game/download/131/BlackDesertOnline885.bcab?beg in=1238753107&end=1247683342&hdnts=st=1537229206~exp=1537230106~acl=/live001/game/download/!/live001/game/patch/~hmac= b99eb03688087d7664b1e5cb69a56c57345dd928a4695dd18257486c2c94cd94" VERBOSE: GET http://akamai-gamecdn.blackdesertonline.com/live001/game/download/131/BlackDesertOnline885.bcab?begin=1238753107&end=12 47683342&hdnts=st=1537229206~exp=1537230106~acl=/live001/game/download/!/live001/game/patch/~hmac=b99eb03688087d7664b 1e5cb69a56c57345dd928a4695dd18257486c2c94cd94 with 0-byte payload curl : An error occurred while processing your request. Reference #199.45b2ef50.1537409669.788a8c0 At line:1 char:1
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebExc
eption
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
should I be running wireshark when the application start the request on the client pc?
Yes. The tool you are good at can easily analyze the problem.
I am getting this now says hit but its not pulling the 8mb file
wget "akamai-gamecdn.blackdesertonline.com/live001/game/download/131/BlackDesertOnline885.bcab?begin= 1238753107&end=1247683342&hdnts=st=1537229206~exp=1537230106~acl=/live001/game/download/*!/live001/game/patch/"
StatusCode : 200 StatusDescription : OK Content : bdoc/| ÐUFK RawContent : HTTP/1.1 200 OK Connection: keep-alive X-N: S X-Upstream-Cache-Status: HIT Accept-Ranges: bytes Content-Length: 13 Content-Type: text/plain Date: Thu, 20 Sep 2018 02:30:17 GMT ETag: "5518bf566... Forms : {} Headers : {[Connection, keep-alive], [X-N, S], [X-Upstream-Cache-Status, HIT], [Accept-Ranges, bytes]...} Images : {} InputFields : {} Links : {} ParsedHtml : mshtml.HTMLDocumentClass RawContentLength : 13
The cached content is incorrect. Delete it and request it again.
still no go after deleting cache i see it trying to fill the cache and this is one of the files in cache directory
^E^@^@^@^@^@^@^@��^Y\^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@Q^D�[^@^@^@^@�^K�[^@^@^@^@�ì�^@^@�^A�^B-"a95d9304b94861f6c3a8367fe93eac54:1537344593"^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@$ KEY: pearlabyss/live001/game/download/132/BlackDesertOnline891.bcab HTTP/1.0 200 OK Server: Apache Accept-Ranges: bytes Content-Type: text/plain Last-Modified: Wed, 19 Sep 2018 08:09:53 GMT ETag: "a95d9304b94861f6c3a8367fe93eac54:1537344593" Date: Thu, 20 Sep 2018 02:53:08 GMT Content-Length: 13 Connection: close X-N: S
bdoc^Ah|^E^@�2)?
[19/Sep/2018:22:12:08 -0500] - REQUEST="pearlabyss/live001/game/download/132/BlackDesertOnline891.bcab?begin=109683834&end=109684249&hdnts=st=1537413088~exp=1537413988~acl=/live001/game/download/!/live001/game/patch/~hmac=a7abcd104bb93d2c54e918247c87f0abf8cd69866751f1a8ec35308834897906" - KEY="pearlabyss/live001/game/download/132/BlackDesertOnline891.bcab" - CACHE="MISS"
Is this request MISS every time? Has it ever had HIT?
when I get it to HIT then the client software does not like it maybe a rewrite inside of the config? since this changes every time "*~hmac=a7abcd104bb93d2c54e918247c87f0abf8cd69866751f1a8ec35308834897906"
also when it hit's it only downloads like a 1kb file when the file is 30gb.
revers proxy nginx with nginx-hmac-secure-link can this be done? https://github.com/nexusofdoom/lancache-installer/issues/20