TitleDB access _MUST_ use a User-Agent header.

[ksanislo]

Please make sure that all access you're directing toward api.titledb.com has an HTTP User-Agent header set. It needs to contain at a minimum the application name and version number that is making the request, something like

User-Agent: Homebr3w/1.0.0

Or more thoroughly, since you may be redirected to a host that won't return results unless it thinks you're a web browser:

User-Agent: Mozilla/5.0 (Nintendo 3DS; Mobile; rv:10.0) Gecko/20100101 Homebr3w/1.0.0

If you don't set a User-Agent header, you're going to be blocked out of access to the API backend until it is added. Right now I'm still allowing unidentified clients since FBI wasn't set up correctly, but the next version has already been patched, and I'll begin blocking unknown clients sometime shortly after it's public release.

[ksanislo]

Alright, so I see that lpp-3ds just doesn't have header support, or can disable ssl cert verification, or the ability to follow URL redirection, or handle chunked encoding (uh... does this do anything network related correctly?)

I'll fork and patch it, and get a new version up that can actually handle HTTP worth a damn before I enforce the User-Agent: requirement...

[Wolvan]

That's the reason why I proxy through my PHP script right now. I thought I had the User-Agent properly set up but it seems I forgot to add the User-Agent: in include.php. The PHP script is mostly fixed in 7256cea78647aeaadb1dc01cfb68f1bfa2534fce but I am still having a bit of trouble with getcia.php.

For some reason, using file_get_contents() on your proxy/titleid endpoint returns the .cia as expected, using the http_req() function that I defined in include.php on the other hand fails and only returns [] as file content. Do you have an idea?

[ksanislo]

Maybe not including the proper path? I see this request come through... There's no API version number in it... (also, you can GET the /v0/proxy/ results... only the main API requires POST)

[12/Jul/2016:11:02:34 -0700] "POST /proxy/0004000000130800 HTTP/1.1" 200 3350 "-" "Mozilla/5.0 (Nintendo 3DS; Mobile; rv:10.0) Gecko/20100101 Homebr3w/1.0.0 Homebr3w-Proxy/1.0.0"

[Wolvan]

That was when I was playing around with settings right now when I was testing locally on my xampp server. None of the changes I made did work though, it always ended up with a 3 byte sized file containing []

[Wolvan]

0004000000130800.zip

These are both files I got with the 2 different methods.

[ksanislo]

Can you fire off a test both ways using this API url instead of the production one, so I can more easily see what is going on in the logs...

https://3ds.intherack.com/titledb-dev/v0/

[Wolvan]

Done. Sent 2 requests, the first one was with get_file_contents and the 2nd one was http_req

[ksanislo]

213.240.84.241 - - [12/Jul/2016:11:30:23 -0700] "GET /titledb-dev/v0/proxy/0004000000130800 HTTP/1.0" 302 4049 "-" "-"

213.240.84.241 - - [12/Jul/2016:11:31:01 -0700] "POST /titledb-dev/v0/proxy/0004000000130800 HTTP/1.1" 200 3350 "-" "Homebr3w/1.0.0 Homebr3w-Proxy/1.0.1"

So, you need to make sure you GET instead of POST with http_req when using the proxy or redirect actions... a GET of /v0/proxy/0004000000130800 is actually taken internally as a POST of /v0/ with JSON data {"action":"proxy","titleid","0004000000130800"}, But keep in mind you can't provide this as JSON externally as the command validator will discard it currently. It really MUST be a GET action right now.

[Wolvan]

Reopening actually, since I have to figure out a method of redirection follow that actually works on my current webserver.

[Wolvan]

For some reason, on my production server it doesn't follow the redirect, only printing out the 302 FOUND message. Works fine on my XAMPP dev server though. Ideas?

[ksanislo]

Maybe it doesn't follow redirects by default? If so, you can capture any 30x return code, pull the "Loacation: http://newurl.com/goes/here" header out, and parse it manually.... but there's probably some kind of option to make curl just follow redirects itself.

[Wolvan]

It doesn't follow redirects by default, no curl does. You have to set CURLOPT_FOLLOWLOCATION to true. Which I did, see

It's weird that it works fine on my dev server but as soon as I put it on my production server it fails to follow the redirection.

[Wolvan]

Files that get proxy extracted seem to get sent through properly at least

[ksanislo]

Well, it probably doesn't matter too much, as lpp-3ds will be able to handle all this pretty soon...

If you'd like to test a new LPP that has been modded to allow headers/POST, there's a compiled test build at http://home.intherack.com/lpp-test.zip

downloadFile has been extended to take up to 5 options, instead of just 2, and the downloadString varient takes 1 to 4....

The extra three in order for each are 'headers', Which is a newline delimited string of all headers that should be in the request... so something like:

"User-Agent: NewUserAgent/v1.0.0\nX-Extra-Header: Value for the extra header" would define both User-Agent and X-Extra-Header

then method, which defaults to "GET" but can also be POST or HEAD (all caps, case sensitive)

and finally the data string to be posted, so { "action":"list" } or whatever

It may segfault and die instantly, I don't have any way to check things yet and I don't really know jack about Lua in general....

[Wolvan]

I'll definitely give it a try later/tomorrow. The PHP proxying really isn't ideal and I wanted to get that done away as soon as possible. Does this check SSL certificates?

[ksanislo]

It disables the SSL cert check, yeah... no URL redirection handling or Chunked encoding support yet though, and it's probably not entirely stable...

[Wolvan]

If redirection is still a problem that means my getcia.php still needs to remain for now but I'll see about rewriting the other information fetches

[ksanislo]

Redirection is entirely fixable... I just need to figure out how to replace the existing URL in the lua string pointer so I can re-call the lookup to the new URL, I just need to figure out how some of these internal functions like luaL_checkstring() and lua_pushlstring() are used to mangle the one lua_State *L pointer that everything gets passed back and forth with. That, and I'm still at work at the moment, so I haven't been able to look through the Lua code very much yet.

[Wolvan]

Take your time, no worries

[Wolvan]

Tried using Network.requestString and Network.downloadFile on my local server, doesn't go through and just freezes the App. No request came in on my server so it looks like the HTTP request is not even executing.

[ksanislo]

Alright, I've got some idea of what might be causing problems, and I'm finally home from work, so I'm going to try to figure out how to set up some kind of debug environment to test and finish this

[Wolvan]

Thanks man, appreciate your help on this a lot

[ksanislo]

Okay... So parsing headers is kind of hanging me up at the moment, but http://home.intherack.com/lpp-test.zip has been updated to my current testing version now... Usage is like:

    Network.downloadFile("https://api.titledb.com/v0/","/Downloaded.txt","User-Agent: LuaShit/1.0.0","POST",'{"action":"list"}')

This supports 301 through 308 URL redirection, file length unlimited by system ram (so things like Haxxagon with BGM that's 100+MB can be installed now) doesn't care about a missing Content-Length: header anymore... basically, everything but headers works now, but only for downloadFile... I need to patch up the downloadString still, and fix header parsing so they set correctly. It's more than enough to give you something to write code against that will actually work though.

[Wolvan]

It doesn't freeze anymore, so that's good. Instead I am getting Error: [string "?"]:<LINE>: error opening url. Code is Network.downloadFile("http://homebr3w.wolvan.at/geticon.php", "/test.png", "User-Agent: Homebr3w/"..APP_VERSION, "GET", json.encode({action = "list"}))

[ksanislo]

That's actually the error kicked out when the server came back with something other than 200 (though it should be anything other than 200/204/206/304) ... Can you check the webserver logs to see what was actually sent back?

[ksanislo]

I've updated http://home.intherack.com/lpp-test.zip with the requestString method patched to match downloadFile... I no longer get any buffer overflow/crashing with either of them, but I can't do any real testing until after work... You can give it a quick run-through and see if you can identify where the problems are, go for it... if it's something simple i can fix it and recompile, otherwise I'll get back to it after around 6PM PDT

[Wolvan]

Server still doesn't get a request. Found the problem of me having reversed path and downloadURL. Unfortunately, requestString still tells me error opening URL and downloadFIle doesn't seem to fire a request at all anymore. No error but the file doesn't end up on the SD card. Also my apache log doesn't show any new hits either. (It does show hits when I browse to it in chrome, so the webserver is working) Something else I noticed is that when I go back to HBL, the Network icon shows disconnected but starting the app doesn't trigger the checkWifi error.

[ksanislo]

Strange... What does it do if you use this for the index.lua script?

-- Check if WiFi is enabled
if Network.isWifiEnabled() then

        -- Download a string, then download a file
        local remoteData = Network.requestString("https://api.titledb.com/v0/","User-Agent: LuaShit/1.0.0\nX-Extra-Header: Extra Data","POST",'{ "action":"list"}')
        print(remoteData)

        Network.downloadFile("https://api.titledb.com/v0/","/Download.txt","User-Agent: LuaShit/1.0.0\nX-Extra-Header: Extra Data","POST",'{ "action":"list"}')
end

-- Return HBL
System.exit()

[Wolvan]

No download happening and printing doesn't work on 3ds anyways. It just jumps to system.exit

[ksanislo]

So, the download call seems like it works fine, I'm not 100% on the requestString method or how to utilize it properly though... Can you figure out how to make a test case that shows what it's actually returning and if it works as expected?

[Wolvan]

Sure! One moment, I'll report back to you

[Wolvan]

https://gist.github.com/Wolvan/5e6a941d30bdea498c6967ea17fc9d68

Have this, still getting the Error opening URL on line 21

[Wolvan]

Commenting out the Network.requestString makes the app run to System.exit() but no file gets downloaded at all

[ksanislo]

What gets appended to /aaa-outfile.txt on the SD card when it does that?

[Wolvan]

File gets created but stays empty

[ksanislo]

Strange... To be created but stay empty it would have to die during httpc context creation... That shouldn't happen unless the OS thinks networking isn't up or something.

[ksanislo]

Are you testing as .cia or as .3dsx? 3dsx may not crash quite the same way

[Wolvan]

I am using the .3dsx build. Want me to try .cia?

[ksanislo]

Just to see if it's different for you, sure, give it a shot and see what happens.

[ksanislo]

I just tried your test script dropped in place of the main file in a clone of this repo to build as a cia (I don't have HBL and it's far less convenient anyway) and I got four .json files and Download.txt, and everything looks like it worked just fine for me.

Edit: Strange... it works on my n3DS, but gives a crash on my 2DS... still trying to figure out how that happens.

[ksanislo]

Alright, Lua Player has been patched, the finished build is at http://home.intherack.com/lpp-test.zip again. Pull request with docs for the changes on it is at https://github.com/Rinnegatamante/lpp-3ds/pull/36 so it should be in mainline shortly.

[Wolvan]

Yes! It works perfectly for the .cia build! Unfortunately, for some reason the .3dsx version seems to be screwy, sometimes returning nil for getJSON instead of a table. I guess I should drop 3dsx support?

[ksanislo]

I'm not entirely convinced that the stack isn't entirely hosed by the point it's doing the download loop for the string mode... The reason I think so is that this should be a u32 however, when it is, the whole thing segfaults (or similar, I don't have a debugger for the 3DS) here in line 285 which is just basic integer addition. I don't have an explanation for why this would be the case, but maybe the default stack is too small (its 32K, iirc) and when built as a .cia, it gets a 256K stack set from cia_workaround.rsf

If you want to try a build with a 256K stack explicitly set, I've recompiled http://home.intherack.com/lpp-test.zip ... The previous good as a .cia version is http://home.intherack.com/lpp-stable.zip in case you need it back for some reason.

[Wolvan]

For now I decided to drop 3dsx support, you can't install CIAs with it anyways so it hasn't much use apart from being no inconvenience to support. .cia and .3ds work fine so those are the ones I'll push in a moment, I just have a few more small things I have to add/fix before doing so. tl;dr: Dropped .3dsx, gonna push a new release later.

[ksanislo]

Awesome... No rush on it. The guys on #3dsdev just reminded me that I can turn my console into a dev unit using the luma-dev fork(?) and might be able to get at least a stack trace out of LPP with that. So I may be able to to fix things properly this evening anyway...

[Wolvan]

Alright, that's good to know. I'm done with what I wanted to add for this version, I'll have my beta testers try it out anyways before making a release out of it, so take your time.

[Wolvan]

Did you take a look at the .3dsx already? Fine if not, just curious

[ksanislo]

Sorry man, I haven't had any free time to dig into it yet. Hopefully I'll have some time to look at it this evening. How is beta testing the new version going, any other problems?

[Wolvan]

That's cool. Both my beta testers gave me an OK on the current feature set and haven't noticed any bugs, so I'd be ready to push a new release on my end.

[ksanislo]

I'm unable to reproduce the crash, so you may as well release without a 3dsx. It may take a while to get it fixed. On Sat, Jul 16, 2016 at 10:50 AM Wolvan notifications@github.com wrote:

That's cool. Both my beta testers gave me an OK on the current feature set and haven't noticed any bugs, so I'd be ready to push a new release on my end.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Wolvan/Homebr3w/issues/1#issuecomment-233142494, or mute the thread https://github.com/notifications/unsubscribe-auth/AECriPvHzSPG7rqjHn5R-0LZ4BTB_lhrks5qWRntgaJpZM4JKlhr .