Audio Support?

[sudofox]

I saw you mentioned on the 3dscapture.com forums talking to Loopy about audio capture. Any news on that front? Would be great to bring it in as an audio input device

[Gotos]

Hi there!

Unfortunately I never got around to figure out why I couldn't get the audio to work at least decently. And I'm not sure I ever will. I haven't touched this code in ages and don't think I will. I don't really have the time to keep developing this anymore. :/

Sorry to bring bad news!

[sudofox]

That's a real shame. I got a capture-modded 3DS to produce support documentation for something called Sudomemo and it would be really amazing to get audio with it too. I think there's still tons of potential for Cute3DSCapture to really expand. Maybe if I get a bit more experience writing C++ I could extend it

[Gotos]

By all means, I'd appreciate pull requests for this! :)

[sudofox]

Good news, I've made some progress figuring out how to decode the audio, bad news, my lack of experience with C++ is slowing me down. I'll keep you updated though!

[Gotos]

Sounds good! I mean, I might be able to help with the C++. Been a while since I last actively coded C++, but maybe our combined understanding might do the trick :D Do you need help with anything specific?

[sudofox]

Please oh please, I would love your help. Here's my notes.

audio research notes

Fired up wireshark for my first ever attempt at sniffing USB traffic.

The URB_BULK in from 3DS->host has byte length of 520615 (sometimes 520519, seen 520623 and 522527) each usually.

Actual Packet Data Length is a bit smaller by 27 bytes.

520615 -> 520588
520619 -> 520592

Screen data should be 518400 bytes each

520588 -(240*720*3 = 518400) = 2188 leftover bytes.
520588 -(240*720*3 = 518400) = 2188 leftover bytes.

The data after the frame contains the audio, though there might be some extra stuff in there screwing it up.

I spent a while absolutely failing at trying to write a uint8_t starting at an offset to file (must've done hundreds of Google searches trying to get this darned thing to work) until I eventually gave up and wrote it to stdout, piped that into a file, and tailed that into ffplay. I was correct! Fiddled with ffplay until I determined that the audio is recognizable (though certainly not good) trying to interpret it as raw PCM s32le. (tail -f extraData.bin | ffplay -f s32le -`)

in main.cpp, this goes after case CAPTURE_OK::

             size_t i;
                for (i=518400;i<(sizeof(frameBuf)); i++) {
                    cout << frameBuf[i];
                }

Also, I tossed in using namespace std; so I could use the cout thing to write to STDOUT.

In 3dscapture.h, I added 2000 bytes as a hack to the FRAMESIZE def in the enum (doesn't affect rendering as that stuff is hard-limited in toRGBA I think)

    FRAMESIZE    =FRAMEWIDTH*FRAMEHEIGHT*3 + 0x7D0,

Anyway, I now get audio. The audio is too fast, really scratchy, but it is indeed recognizable. Maybe bitrate or encoding is wrong, or maybe I'm reading the wrong amount of data (more likely, and as PCM in certain forms often uses data from previous samples to choose how to correctly play the next sample, maybe part of the issue)

[sudofox]

Here's a YouTube video showing what I managed to achieve. Sorry for the phone recording, my screen-recorder doesn't want to do audio right now.

https://www.youtube.com/watch?v=LZxUwIwCtI8

Reviewing my footage, I'm thinking the slowdowns/speedups might have to do with how quickly the audio data can get to ffplay as well since I'm probably using a really inefficient method of spitting it out to stdout. catting the resulting file to ffplay plays it back really fast, maybe limiting the sample rate would slow it down to the proper speed.

There's probably some extra data or missing from the USB packets (almost certainly I'm reading some garbage) and I might try limiting the length of the extra data I'm reading to a rounder # of bytes, maybe i'll look in the file with a hex editor to see if i can find any obvious offsets?

[sudofox]

also idk how useful this will be but

Wireshark filters for only data sections (multiple addresses so we regex-select the usb source):

((usb.src ~ "1.5..") || (usb.dst ~ "1.5..")) && (usb.endpoint_address.direction == IN) && (usb.data_len > 100)

1.5.x was just what it addressed out to on my windows box. After I poked around in it for a while i switched back to developing on my linux box.

I probably could've gotten more done but I was doing most of this on my lunch break

[Gotos]

Going by what loopy states, what your doing to get the audio data should be correct - just read more data after receiving the frame. That's how far I got back in the day as well. The tricky part in my eyes is getting the buffering and sample rate and stuff right. The sampler rate should be 32728Hz, most playback tools will assume 44.1kHz if you don't specify anything else, cause that's by far the most common. Jittering might be caused by either wrong sample rate (I assume in that case the sound would also sound higher pitched, at least if the 33kHz get interpreted as 44kHz) or not enough buffering - if I'm reading your comments correctly and if my memory is correct, the 3ds capture unit doesn't always send out the same amount of audio data, right? So with wrong assumptions about that, you could have an empty buffer.

I haven't watched your video yet and unfortunately can't right now, so if any of what I said didn't make sense and the video would make that clear, I'm sorry :D

Hope this helps!

[Edit:] I just noticed the parameter you passed to ffplay - I assume that one sets the sample rate? In that case it seems close enough that I would assume that the sample right would only introduce minor artefacts... But piping the data through via pipe might not buffer the data enough if you're not holding an internal buffer, so maybe it's a buffer underrun

[sudofox]

Wow, that was super helpful! I tried to force the bitrate with the -ar param and it worked wonders!

(note that audio data is being sent to stdout)

 ./Cute3DSCapture | ffplay -ar 32728 -f s32le -

There's still a lot of buzzing and hiccups but the audio sounds a lot better now. I'll try improving upon it later today, only have a second to test right now.

The -f s32le sets it to interpret as signed 32-bit little endian PCM.

[sudofox]

I did try stripping out the large long sections of null bytes that I was getting in the stream by sticking this before ffplay:

./Cute3DSCapture | perl -np -e 's/\0\0\0\0\0\0\0\0//g' |ffplay -ar 32728 -f s32le -

It improved the skipping and lag a little bit more. For some reason I'm seeing the number 0x900 come up here and there looking at my hex editor?

I'm pretty certain that this can be solved by grabbing the correct number of bytes from the right offset and excluding any extra junk. Also might be possible that the length of the audio data is included in the output data, perhaps before the PCM data starts?

If Loopy could weigh in on the exact format of the USB packets, we'd have this in the bag.

[sudofox]

I haven't had the time to look much more into this but I did throw together something on my phone that barely works: https://github.com/sudofox/Sudo3DSCapture

I may use it to experiment with the audio stuff. I'm super new to all of the things used in this so I could use some input getting it into a working state :D I wrote it from my phone so the code quality is pretty bad. Oh well!

[Chromagram]

Any progression on this by chance? I don't have Windows. CuteCapture does wonders in capturing the video, but with no audio I can't use it for livestreaming as intended. I've tried running loopy's Windows app through Crossover, WINE and Parallels, but I either can't get it to work at all (USB limitations) or OBS won't pick it up.

Really hoping this audio support continues development, and thanks for the work!

[Gotos]

No progress on my end and as far as I know sudofox got stuck at roughly the same part I got stuck at back in the day.

I usually record the audio right off of the audio jack, connecting the 3DS audio jack to my audio interface. I think there are ways to do that without an audio interface as well, if you run that cable into your computers line in? In that case you'd need to reroute the audio coming into your line in to the output you're capturing with OBS (the one it considers desktop audio) so that obs would capture the audio and you would hear it too.

I know it's not a great work around. But unfortunately I won't have the spare time to look into real audio support myself in the near future :(

[Chromagram]

No progress on my end and as far as I know sudofox got stuck at roughly the same part I got stuck at back in the day.

I usually record the audio right off of the audio jack, connecting the 3DS audio jack to my audio interface. I think there are ways to do that without an audio interface as well, if you run that cable into your computers line in? In that case you'd need to reroute the audio coming into your line in to the output you're capturing with OBS (the one it considers desktop audio) so that obs would capture the audio and you would hear it too.

I know it's not a great work around. But unfortunately I won't have the spare time to look into real audio support myself in the near future :(

Thanks for the reply, I don't mind the workaround, but I can't seem to get it to work with the audio jack. I've tried two cables that have 3 rings for microphone, and I'm using Loopback to route the audio. However, the Mac doesn't pick up the plugged in 3DS as an audio source at all. Any ideas on how to resolve this? Thanks!

[Gotos]

Do you mean a TRRS connectors like this one? https://en.m.wikipedia.org/wiki/Phone_connector_(audio)#/media/File%3A3.5mm.jpg

If so, that's one ring too much. Typically, the four sections of TRRS connectors are used for ground, left audio signal, right audio signal and mono microphone signal - in this case, we work with a stereo audio source that doesn't have a microphone tho. So you want to use a TRS cable.

However: I'm not all that familiar with Macs. I think the try to detect if you're plugging in a TRRS cable or a TRS one and will always try to use TRS as output - when you connect the 3DS this way you want that jack to be used as input (more specifically: line in) tho. You might have settings to your Mac to use that jack as line in. If so, that's what you want to do (plus using TRS cables). If you can't manually configure that, I'm not sure what do to tho :(

[Chromagram]

Yes that’s the one I have. I also do have a TRS cable but couldn’t get that to register either. After the TRS cable didn’t work, I read up online and saw that people suggested a TRRS cable. The issue with Macs is that they don’t have a separate jack for headphones/mic, so you only have one jack. I couldn’t get either one to play any sound. I’ll try again with the TRS cable, but I couldn’t find a way to have the Mac show it as an input, just output as headphones.

[Gotos]

That's unfortunate. If you can't configure the jack as line in, you might be out of luck :(

[sudofox]

No progress on my end and as far as I know sudofox got stuck at roughly the same part I got stuck at back in the day.

I think we should ask Loopy for help at this point, maybe he'll be friendly :)

[Gotos]

He definitely is! I did get some pointers from him back in the day.

I'm not currently interested in working on this myself, but if you are, do feel free to do so and send a PR :) And again, I'm sure loopy would be willing to help, if you reach out to him :)

[Chromagram]

Revived potential in this would be amazing! I'm currently at the point of digging through the internet for one of Loopy's old 3DS controller mods so I can just leave the 3DS charging and plugged into a separate PC to capture it while using something like a wireless WaveBird controller. Anything to get the picture/audio on my Mac without a set of long wires running across my floor.

Excited to see where it goes :)

[nealt]

(this is loopy) It looks like you've got it pretty much figured out. Some pointers: There shouldn't be any junk data. Everything following the frame is raw audio. The size isn't fixed, just read until the stream stops. My capture app tries to read an extra 8k, but you should normally get back closer to 2k (~2188). As you've noticed, the sample rate is around 32728Hz. I'll duplicate or drop a sample as needed to keep in sync with playback.

[Chromagram]

Thanks loopy for looking into this. I'm really hoping something usable for Mac materializes, my 3DS capture board is just collecting dust now.

[spitford]

I don't know if anyone is still actively working on this at all, but I found out subtracting 324 from the sizeof(frameBuf) gets rid of the motor boating. It's still pretty static-y, so I'm sure that the 324 isn't the best number, but it doesn't completely destroy my ears, and at least gives us a place to start!

in main.cpp, this goes after case CAPTURE_OK::

          size_t i;
             for (i=518400;i<(sizeof(frameBuf)-324); i++) {
                 cout << frameBuf[i];
             }

I am also using 0x7D0 as the hack to increase the frame size.

    FRAMESIZE    =FRAMEWIDTH*FRAMEHEIGHT*3 + 0x7D0,

[Chromagram]

If even a tiny bit of progress happens on this I'll hold out some kind of hope, since I have to run Windows on a VM just for 3DS Capture. One day I'll be able to fully ditch Windows and stop being assaulted by this terrible OS.

[spitford]

Ok, so I've been able to get a clean-ish audio sample out to ffplay, but the audio quality drops as time passes, and it gives out eventually. I'm not sure if this is a limitation of ffplay or me not dropping/syncing the samples I'm getting. I have to figure out how to get SFML to play the samples, as I assume that's what was planned. @Gotos any chance you have an idea of how to play the audio?

[Chromagram]

Ok, so I've been able to get a clean-ish audio sample out to ffplay, but the audio quality drops as time passes, and it gives out eventually. I'm not sure if this is a limitation of ffplay or me not dropping/syncing the samples I'm getting. I have to figure out how to get SFML to play the samples, as I assume that's what was planned. @Gotos any chance you have an idea of how to play the audio?

Heya, thanks for continuing to work on this, that's really awesome. I'd see if you can ping @nealt as that is loopy who designed the board as well as (iirc) developed the Windows version of software that does both audio/video in an easy single cable. loopy's own forums at 3dscapture.com also is a really good place to reach out to him for a second set of eyes looking in. I'm sure he has a lot on his plate, but he's also very helpful in replying.

[Catwashere]

Sorry I cannot help with this because I have a N3DS, but mainly the issues with the noises is because you are probably adding twice a same audio buffer, possibilities chip send it duplicates or you are processing it multiple times, you are adding bytes to audio that are not for the audio and because the sound is taken out though weird processes for ffplay. I see here you are using also sfml, so feel free to take the SoundStream and the conversion of buffer to it from my PR for N3DS.

For N3DS the 2188 bytes is kind of true, sometimes is 2196 or less in my test is weird it goes up

Cheer and give credits, feel free to contact in case of doubts