Figuring out Unity Integration

[diemildefreude]

I have a somewhat complex PD patch that I want to integrate into Unity via Heavy. I've simplified it here while still leaving a lot of the objects required for the full version of the patch. I also thought the sample loader might be the problem, so I've detached the related abstraction, [gs], and replaced it with an [osc~]. If you unzip "osctest070916.zip" and open _main.pd with Pure Data, you'll hear that it works: A held sinus tone with periodic fluctuations in amplitude (the length, form and steepness of the amplitude envelopes change as you increase the value next to "passage staircase", [r passage @hv_param 0 100 0].

Uploading this .zip into Heavy and placing the Unity files into the Assets and Script directories of the project, I press play and nothing happens. I've exported a package of the project: "osctest_unitypackage.zip"

osctest070916.zip osctest_unitypackage.zip

[diemildefreude]

Update: I tried this with a super simple patch [osc~ 300]>[*0.1]>[dac~] and I still ge no sound.

[diplojocus]

@diemildefreude hey sorry about the delay! I've been on my mac recently so not being able to correctly test your project.

That simple patch you got there should definitely work! Promise I'll take a look this week!

[diplojocus]

Oh have you made sure that the LibWrapper plugin has the correct platform settings enabled in Unity?

It should look like this:

[diemildefreude]

Hey Joe, Thanks for the answer. It looks like those settings are fine (see below). Unity's audio playback is working... if I deactivate the Wrapper and set an audio file to "play on awake", I hear it just fine...

simple.inIj0.unity.win.x86_64.zip

[diemildefreude]

simple.zip

The super simple patch is in there. I assume I don't nead to [loadbang]>[; pd dsp1;{ for Heavy...

[diemildefreude]

Ah, and the previews in the Heavy Uploader don't produce any sound either....

[diplojocus]

hi @diemildefreude I just uploaded that simple.zip file to my own area and it seems to work for me: https://enzienaudio.com/h/diplojocus/sine/

I also created a test project in Unity 5.4.0f3, imported the Unity build of the simple osc patch, created a empty gameobject and attached the libwrapper script, on pressing play I got sound as expected, I'm not entirely sure what's going wrong for you.

Can you send a link to your patch on the heavy site? And maybe explain a bit more the steps you're taking?

I did try out the osctest unitypackage you uploaded and that did crash Unity so I'll take a look into that!

[diplojocus]

hey @diemildefreude, ok so the first thing I noticed is the rhythm abstraction's metro was dependent on a toggle box that initialises itself on patch load in Pd, which I think is a recent feature added right? Anyway I wasn't aware of it before.

This isn't something that we support in heavy. Essentially what happens with toggle and number boxes is they are turned in [f ] objects (I think the toggle will even have a [!= 0] on its input as well. GUI bangs for example also get turned into [b ] objects. So you can imagine that a lot of the features in the GUI side of Pd, incl object properties, won't be supported in the heavy output as it mainly focuses on audio and control-rate information.

Sorry that it was non-obvious what the issue was there.

[diplojocus]

Ok so I found some other GUI objects, like sliders and radio boxes in your patch that are relying on being initialised on load, and I think this is in turn causing the patch not to make any sound when playing in heavy. I think it would be worth going through your patches and [loadbang]'ing everything so that the default are definitely set in heavy.

I would also suggest, if you can, to go through your patches, try to simplify the logic and clean out any redundant code, as it wasn't very straightforward to determine the source of the issue.

I'm gonna mark this question as solved as right now it doesn't look like it's an issue in Unity.

[diemildefreude]

hi @diemildefreude I just uploaded that simple.zip file to my own area and it seems to work for me: https://enzienaudio.com/h/diplojocus/sine/

I also created a test project in Unity 5.4.0f3, imported the Unity build of the simple osc patch, created a empty gameobject and attached the libwrapper script, on pressing play I got sound as expected, I'm not entirely sure what's going wrong for you.

Ah, sorry about that. I didn't have Play on Awake checked. The simple patch works now. Oddly though, the "sine" preview still doesn't make any sound. I wonder if I need to update Java or something...

ok so the first thing I noticed is the rhythm abstraction's metro was dependent on a toggle box that initialises itself on patch load in Pd, which I think is a recent feature added right? Anyway I wasn't aware of it before.

This isn't something that we support in heavy. Essentially what happens with toggle and number boxes is they are turned in [f ] objects (I think the toggle will even have a [!= 0] on its input as well. GUI bangs for example also get turned into [b ] objects. So you can imagine that a lot of the features in the GUI side of Pd, incl object properties, won't be supported in the heavy output as it mainly focuses on audio and control-rate information.

Ok so I found some other GUI objects, like sliders and radio boxes in your patch that are relying on being initialised on load, and I think this is in turn causing the patch not to make any sound when playing in heavy. I think it would be worth going through your patches and [loadbang]'ing everything so that the default are definitely set in heavy.

The toggles, radio boxes, etc. are mostly not self-initializing, but are all inititialized by a single loadbang>[message( in the [pd setup] patch above the "rhythms" abstraction (see below). This is nothing new in PD (it's used in a lot of help files). Is it really not supported in PD? Of course, I could have a million separate [loadbang]s sending a single message into a single [send] object, but is that really necessary..?

Otherwise, it looks like there were indeed a few sliders in the "reverb" abstraction　and one in the "gs| abstraction with saved input (when "properties" were opened, the button displayed "Save" instead of "NoInit". I don't think this feature is that new either, but either way, the sliders should still receive commands from [pd setup]. I'm uploading a version now with no "saving" of slider values.

I would also suggest, if you can, to go through your patches, try to simplify the logic and clean out any redundant code, as it wasn't very straightforward to determine the source of the issue.

I would love nothing more, Joe, but it's possible that I don't see the redundancies a more skilled programmer might. I know you don't have time to help me redo all my patches, but if you could point out one or two examples of what you mean, I'd be much obliged.

[diemildefreude]

OK, I removed all the abstractions and added in just the panner. Indeed, using a single message to send out commands to multiple receive objects seems to not work in Heavy. They seem to each need their own [loadbang](see below). When played in Unity, I hear the sine wave, but no panning effect. Is this because Unity's own spatialization effects supersede those of Heavy/PD?

https://enzienaudio.com/h/sep/osctest/JbnE6/

osctest160916b.zip

[diemildefreude]

In any case, since it was half-working, I went ahead and tried re-adding the "dynamics" abstraction using individual loadbangs for everything. There are no errors, but also no sound in Unity:

osctest160916c.zip https://enzienaudio.com/h/sep/osctest/

[mhroth]

@diemildefreude You're right that remote messages don't (yet) work in Heavy.

Heavy will output a warning that looks like e.g. [msg remote local] in "_main.pd/setup" @ (x:32, y:104): Message boxes don't yet support remote messages. These messages will be ignored.

Basically, all of the messages in your screenshot won't be processed. If there a ; in your message box, Heavy will ignore it.

[diemildefreude]

Yes, I see that now. These warnings will likely explain why I'm not getting any sound from the last version I posted. I'll comb through them and see if I find any hints. If I hit a wall there, I'll come crying to you again. Hahah...

[mhroth]

We'll be here to give you a hug...

[diemildefreude]

Hey, I need a hug.

I've fixed some small things to get rid of warnings and removed most of the subpatches from the main patch, since these aren't needed to initiate the sound. There are only 5 warnings now: https://enzienaudio.com/h/sep/osctest/

2 for the remote messages: Though they're disconnected, I've left them in the patch to remind me what messages I need to send to other objects. For the objects currently in the patch, I've sent messages to each receive object individually: [pd setup]

2 for [change] The [change] object apparently has a hidden [route] inside of it. However, this shouldn't be relevant for the malfunction, as I only ever send floats to them.

1 for [delay]

When converted with Unity, the plugin still produces no sound in Unity and I've no idea what to do next. Help!

osctest170916b.zip

[diemildefreude]

Yeah, just double-checked again. To summarize: Play on Awake is checked, the plug-ins' platform settings are correct, the patch plays in PD (sine wave with volume and panning changes), uploader only shows those 5 innocuous-seeming warnings: No sound in Unity upon pressing Play.

The only thing I could think to do next is start re-building my patch/abstractions from scratch to be simpler, and, while reintroducing the complexity, periodically uploading them to see if they work. I don't even know if that would work though.

[diplojocus]

@diemildefreude ok i think I see where the issue is. There's various GUI objects (sliders, toggles, etc..) that use the send/receive symbol functionality, (i.e. accessed from right-click object > properties).

Just for purpose of avoiding further confusion, you can assume that any GUI object in your patch will be replaced with only a [f ] object. That is to say all the functionality of the GUI object that you can do in Pd is not supported in heavy. We try and replicate as much as we can but that is one area we don't support.

So you can imagine that if you replace every one of those radioboxes and sliders with a float box, your initialisation system isn't going to work in heavy. You'll need to explicitly add a loadbang somewhere to start things off.

A concrete example is in the dynamics.pd file, there's a receive r $0-envtoggle that is never going to receive anything in heavy as the corresponding send is not triggered (in the Pd patch it comes from the radiobox). This results in the line~ object not being initialised and thus you'll hear no sound.

Hope that helps!

[diemildefreude]

Hey, thanks for your reply, Joe; it helped a lot. When you told me that before, I thought you were talking about the "save/init" option and I failed to think of the other implications of the toggles etc. being changed to floats. I've fixed it and it works with the [rhythms], [dynamics] and [panner] abstractions just fine. But when I introduce the [reverb] abstraction, it leads to Unity crashing.

There is one new type of warning in the uploader:

hip~.pd in "_main.pd/reverb.pd/vfreeverb~/diffusor-right/hip~.pd" @ (x:156, y:684): [rzero~] accepts only signal input. Arguments and control connections are ignored.

https://enzienaudio.com/h/sep/osctest/

osctest210916.zip

EDIT: The hip~ objects are in [reverb]/[pd freeverb]/[pd diffusor-left] AND [pd diffusor-right]

It looks like maybe too much CPU is being used: https://enzienaudio.com/h/sep/osctest/INhEm/ Not sure why this would be, as, when I run the patch in PD, the load meter shows between 0 and 11 CPU and the Windows Task Manager shows about 1% CPU....

[diplojocus]

@diemildefreude nice one!

Ok so you're on Windows right? We literally just pushed a fix for a windows crash to the dev channel today that I'm totally guessing is probably also related to your issue (from a quick glance in your patches).

These changes will end up in the next release, but if you can't wait you can always use the python uploader script and choose to compile from the dev channel (cutting-edge changes).

See the documentation here about how to the use the script https://enzienaudio.com/docs/index.html#3.uploader

After installing the dependencies you'll want to run something like:

python uploader.py /path/to/osctest210916 --out /path/to/output --name osctest --release dev --gen unity-win-x86_64

uploader.py: the uploader script /path/to/osctest210916: path to the directory containing your pd files (incl _main.pd) --out /path/to/output: path for where the requested files are downloaded to, can set this to be Unity Assets folder for example --name osctest: name of the patch (needs to be the same as one on your heavy profile page) --release dev: compile with the dev channel --gen unity-win-x86_64: request Unity Windows 64bit binaries

[diplojocus]

There is one new type of warning in the uploader:

hip~.pd in "_main.pd/reverb.pd/vfreeverb~/diffusor-right/hip~.pd" @ (x:156, y:684): [rzero~] accepts only signal input. Arguments and control connections are ignored.

Yeah sorry you can ignore that, it's not your fault!

[diemildefreude]

Yes, I'm on Windows. In case you didn't see my EDIT in my last message: Do you think this fix will resolve the CPU spike?

When will the next release be? I'd like to do it with Python, but I actually wasn't able to figure out how to last time I tried, even with Martin's help:

Hey Martin,

I haven't been able to get any commands to work in either command line. You can see in the Python window, for example, that "python get-pip.py" doesn't do anything (does it have to be in a certain folder? It was in Downloads). It also doesn't work in the Windows "Eingabeaufforderung" (That's the Windows Command line, I bought this computer in Austria). In any case, I just double-clicked on the get-pip.py file itself and seems to have installed itself (it opened a python window detailing the installation).

As you can see in the Eingabeafforderung window, the uploader command still doesn't work.

On another note, when I double click on the uploader.py, it briefly opens a window that then closes.

I probably need to look up a tutorial on basic Python usage. But if the next release is coming soon, I might just focus on debugging other abstractions until then.

[diemildefreude]

[diplojocus]

hi @diemildefreude the next release is planned for sometime next month.

I think I see where you might going wrong, firstly you're generally best running the cmd app in administrator mode. Right click the icon and you'll see the option.

Also with python, am I right in thinking that you double-clicked the python.exe app to run it? Because there are two ways to use it. If you double-click it (or run it by itself) then it'll run in interactive mode, it'll kind work like pd where you can add commands and do stuff in it. But the other way is to just pass it a file and it'll execute that directly.

Maybe try following these steps:

• open just the cmd application as administrator
• then run: C:/Python27/python.exe C:/Users/yourname/path/to/get-pip.py

Remember that cmd doesn't know what 'python' means unless you tell it to be C:/Python27/python.exe, but you can also add that path to your environment variables so you don't have to type the full thing each time.

See here for more info: http://stackoverflow.com/questions/4750806/how-do-i-install-pip-on-windows#12476379

[diemildefreude]

Alright, I did my best to do everything you said, @diplojocus ... After running as an Admin, making sure pip was up-to-date and using it to install requests, I receive new errors. (I still don't know what "sudo" means in the Heavy documentation, but pip seems to be installed anyway, so I guess it doesn't matter)

[diplojocus]

nice one @diemildefreude!

I can see you're very close, it looks like the last bit is to make sure your uploader script is up to date (I think you've got an older version).

Replace your current uploader.py file with the contents of this page here: http://enzienaudio.com/static/uploader.py

[diemildefreude]

Hey @diplojocus. It worked! Thanks for your patience.

Now comes the tricky part of integrating the granular synthesis module that uses a wav file. In part, it's tricky because I can't test and hear if it works in PD, as I need to load the .wav via Unity.

So a while ago you sent me a SampleLoader.cs file. I'm not sure if it's still up-to-date or how I'm supposed to use it in in conjunction with the patch's script. I've modified it with the name of the patch's script and the .wav file, but it still has no definition for "FillTableWithAudioClip". Attached are the original script and the audio file. A link to the patch is below.

audiosample.zip

sampleloader.zip

https://enzienaudio.com/h/sep/station/

[diemildefreude]

Oh, and as for the patch itself, I also don't know if it's set up right to receive the wav file into its array. If you open up [gs], you'll see near the upper right area a receive object for the table size (visible in Heavy, as seen in the screenshot above) and the table itself. If you need to compare it with the version of the abstraction that works in PD, I've attached that here. In this case, the table/buffer-related items are in the lower left area. Please let me know if you need any other info...

gs_pd.zip

[diemildefreude]

Hey, I checked to see if you'd added any documentation to your website about sample loading and you did! I updated the script according to what I found there, so the errors are gone:

N.B. The table name is 7-buffer, because the [gs] abstraction has 7 as an argument in the main patch uploaded to Heavy.

There's no sound, so, of course, I'm either doing something wrong in Unity or in the patch. I went ahead and outlined in green the main difference between the PD-functional and the Heavy version of the patch in my previous post. I'll keep thinking about anything else I may have missed...

[diemildefreude]

Aw man, I was still holding out for an answer. ;)

On 14 Feb 2017 20:57, "Joe White" notifications@github.com wrote:

Closed #124 https://github.com/enzienaudio/heavy/issues/124.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/enzienaudio/heavy/issues/124#event-961239647, or mute the thread https://github.com/notifications/unsubscribe-auth/APUBNlOiqZzh9MgzuUW3LgJuE6j_Pi82ks5rcZakgaJpZM4J2hzj .

[diplojocus]

sorry @diemildefreude I (mistakenly) thought this got resolved. The error kinda evolved over time! Would you be able to create a new ticket with an example patch that describes the issue?

[diemildefreude]

Hi @diplojocus , I'd love to, but github isn't letting me upload .zip files made with WinRar.... it used to, and I'm not sure what's changed. Otherwise, I can e-mail you directly.

[diemildefreude]

OK, I've made a new ticket: https://github.com/enzienaudio/heavy/issues/178