Open rasputozen opened 9 years ago
rasputozen
commented
9 years ago Right now I'm thinking there must be a file located somewhere that caches the remote scripts, and then I can delete this file to refresh the scripts so they recompile... anyone have any idea about something like this?
thomasf
commented
9 years ago I think they might be compiled to pyc and then loaded into RAM the first time they are imported which is kind of usual for python applications. You probably have to restart live or possibly implementing something in python that allows reloading of the files.. I always restart live and delete any pyc files when I am working on live scripts. I really dont know how this works,, I just gave you my thoughts and procedures.
thomasf
commented
9 years ago Btw. this is my restart script for development (osx)..
https://github.com/thomasf/dotfiles-thomasf-xmonad/blob/master/.bin-xmonad/restartabletonlive-dev
aumhaa
commented
9 years ago FWIW, I've spent a lot of hours trying to figure out a way to rebuild Live's database when reinstantiating a script without success. AFAICT, Live's Python implementation doesn't do things the same way that a standard Python does. Unless there is a hidden switch/method in Live's C code that can be manipulated, standard Python functions like reload() don't work as expected inside Live's Python. There's just no way to reload a script's definitions without manually restarting Live. I'd absolutely LOVE to be wrong about this, but I don't think I am. I've resorted to using an OSA script like thomasf to force quit Live and then restart/recover the set.
a
datamafia
commented
9 years ago I can confirm what @aumhaa just said. I have several thousand restarts attributed to Py script writing in Ableton and tried every tool in the garage to avoid restarts.
I generally bash script deletion of *.pyc files and sometimes set up a task runner (like grunt) to do some work.
To add to the topic - if you watch the folders/files on startup you will see first a compile stage (.pyc created). Once the .pyc is created you can delete the .py files and all will (should) run. This empowers lameness from companies who write scripts and don't want to drop the native. I have received .pyc for scripts - super lame.
So - putting all of that into the mix, yeah, there is weirdness that is Ableton.
Some more information on Py in Ableton. Before leaving the music scene completely I was a royal pain in the ass for Ableton (they need more people on their shit IMHO). My digging, pestering, and extracting information I should "not know" (like from a buzzed Henke at a bar) from various team members and vendors has yielded many interesting tidbits. The piece valuable to this discussion is that Ableton uses a modified complier (custom). I don't know the exact details or differences, but it was made clear to me that there was a need for features or functionality that led to the absorbtion of the compiler into the application (beyond my skillZ) where there is a binding to the C/C++ work(?). Compilers are NOT my speciality, and my C/C++ chops are "meh".
Unless things have changes (and I doubt it) the *.pyc files are always created on startup. There are a few interesting apps out there to spit out logs/inspect, but a simple (linux) tail and a few low level tricks really help you see wtf is happening.
Only reason I am answering is b/c I saw @aumhaa in my emails and thought "not heard form him in a while" so - HI!
thomasf
commented
9 years ago Yeah, it's really sad that ableton has this framework which alot of people use + Ableton almost support people doing things with these APIs (they mention that this is done on their site) but without providing their parts such as documentation and an official license. One can just hope that the relese of bitwig will engourage Ableton to do the right thing.. A possibility (for me) is maybe to switch to bitwig when their modular stuff is finished since I really would like to create compound channels/custom busses of sorts and I do hope that things like that will also be possible with bitwig in the future.
aumhaa
commented
9 years ago @dm: hey buddy :) soooo, why did you check out of the music scene? gotta hear the Henke story at some point (or maybe I did and forgot?) ;) You should drop me a mail and tell me what you've been up to!
*.pyc ARE created at startup, as Live now parses the entire MIDI Remote Scripts folder and builds what it can base on their content. However, if a script isn't compilable when Live loads, it WILL generate a .pyc at the first opportunity it has upon trying to load the script (e.g. the script is broken on Live launch, but a later edit makes the script valid after load and the script is initiated via the MIDI Preferences screen).
This wasn't a problem before Live 9.1, because a "bug" was causing the Python engine to rebuild all of it's module definitons every time a new script was loaded or removed. It was simple enough to delete all the old defs at ControlSurface.disconnect() and Live would rebuild them when it reloaded a script. I was very bummed out when this stopped working.
The big problem here with using a rollback importer model (which I've tried, ad nauseam) is that the main module defs are held at a higher level than the individual control scripts scope access to, so there's no way to reach them to redefine with reloads. It might be possible to get to them via GC or something, but I just don't have the time to spend on it.....I literally wasted like 30 hours when they made the big update several months ago trying to get things to work again, and eventually realized I was wasting more time than I would save if I got it working again.
Also, just for grins, I tried building a C-based module in Juce to speed up updates faster than the 60ms bottleneck (to get around LivePython's custom threading, so we could do clip-chopping via Python), but of course without the actual LivePython source it wouldn't work properly. Of course, some things can be overridden by importing an actual vanilla Python installation, but that's no good for portability's sake.
@thomasf: I can understand their reasoning, to a point. Especially with the Live9 _Framework codebase, there are some geniuses turning out the stuff that they're using at this point (although, FWIW, maybe one too many geniuses, since I often see some divergent methodologies incorporated in different scripts which would be better served by a bit more cohesiveness....but, hey, I live in a glass house so I shouldn't throw stones). Since the majority of the newer _Framework was built to support THEIR hardware, it's probably difficult for them to see it "profitable" to just give it away for others to use freely. We just need to be grateful that they haven't closed it completely (something that would be fairly easy for them to do, I think).
RE: Bitwig. I love the openness of it. It's good stuff, all around. My only problem with switching over is that I have such an enormous amount of time and work based in Live/Max.
Honestly, with upcoming changes to Max, I may ditch use a DAW in my live-rig all together someday ;) Maybe that's the new frontier.
thomasf
commented
9 years ago Well. the biz case (IMHO) would be the same as always has been in this business, catering to engineering needs of studio customization and/or artistic expressiveness. Hell, some people still build their own mixing decks. Integrating flow the way you want using cables and electrical switches/analog mixers as your API still beats closed down environments like Live in some aspects. Another thing to consider is that Abletons age as an system architecture might be showing clear signs of stagnation, possibly a combination of technical issues and intentional product design. Most of their improvements in the recent years has been horizontal rather than adding new functionality that makes new core workflow possible. Max still feels like to much of "something else" for me to consider being that thing, from what I understand it can mostly control aspects of live rather than change behaviour. I also have a lot of time invested in Live but like you I think I am seeking something thats modular at the core.. I have done some smaller systems using supercollider which are limited but fine. Since I never got into max/msp I'll build my analog modular synth hold off the software side of things to see what the bitwig guys come up with.
This is possibly not the place to discuss this, so I'll stop here :)
dm
commented
9 years ago Hey @aumhaa, I believe you mentioned my username by mistake, cheers :)
rasputozen
commented
9 years ago @thomasf if this isn't the place to discuss this, please tell me where and let's move it over there... lol
Seriously, thank all of you guys for these comments... way better than anything I expected.
From what I've read here... I don't think I'm going to reverse engineering these scripts... I literally can't convince myself to reload live every time I tweak a script... maybe if Live was the only game in town.. I was messing around with Bitwig last night actually and I could be making a judgement too early but the open controller api, from what I've seen, has just about sold me on switching my workflow over. Well that and it's basically a clone of live with improvements - though I really miss beat repeat :( There's probably still a few things I'll use Live for, but bitwig is looking really promising.
@aumhaa I haven't really gotten too into Max... played around with PureData for a while.. but I didn't know it could be possible to create your own modular DAW out of it. I'll definitely have to check this out. Do you know any resources to get an idea about this kind of setup?
thomasf
commented
9 years ago @rasputozen I meant this isn't the place for us to blame Ableton for their decisions however valid I feel the criticism is.
Btw, I'll also think its best that you read up on how git basics in a tutorial or something, the magic happens in a hidden folder named .git in the root of the repository and you can always see the status of changes using "git status" but you really have to understand the basic workflow and then you will understand what you were asking about git.
Btw. are you aware of https://github.com/hdavid/Launchpad95 ?
thomasf
commented
9 years ago Also, this repository currently is a decompiled scripts here are from ableton 9.1.4 and the latest version is 9.1.5 so problems you have might also stem from that as well. I havent ever tried to use these directly so I wouldn't know. Also aumhaa's and other people have done great work in https://github.com/LividInstruments/LiveRemoteScripts which I currently am using for a number of unpublished things.
rasputozen
commented
9 years ago @thomasf you know what... I totally understand how the pull request would work now... it just sunk in.. lol - thanks for the refresher regardless.
and I wasn't aware of launchpad95 - good idea to develop an open source LP script.
Yah I see where you're coming from not turning this into a "blame ableton" type thread. I don't mean to be condemning of ableton, all around it's probably the best DAW on the market right now. It would still be awesome to see a correctly decompiled repo of the native remote scripts. Knowing my limitations as a programmer though, the recompile lag is a bit much for me. Hey, maybe ableton will change that back to pre-Live9 functionality? Or maybe they'll release some kind of python console for script development? Or one of us will figure out how to reload the scripts some other way? I guess it's possible.
datamafia
commented
9 years ago This will be my last post on this thread. Hit me up on email/twitters/etc to talk more - esp you aumhaa - I will connect the dots for you.
@aumhaa I checked out mostly b/c Ableton turned out to be a very bad partner and b/c I like programmers better than most musicians. So many artists were getting screwed by Ableton plus they stopped supporting the pros/hardcore users, and I constantly caught the company w/pants down around the ankles in the areas of community, code, QA, planning, and product direction.
As for henke - yeah, that is funny shit. I got a whole bunch of other stories...none make Ableton look good.
For the record : This repository, created by @gluon (who is super cool) is very dubious. My inquiries regarding certification (they kept inviting) and Python scripting for Ableton (among many questions) and the answers contributed much to my leaving the scene and fueled a vitriolic disdain for Ableton and everything they do.
Business...
aum - you say "Live now parses the entire MIDI Remote Scripts folder" -- strike "now", live 8 and 9 parse/compile nearly identically afik (form a spectator view). I can accept wrongness if necessary.
The MIDI Preferences screen merely "moves the pointer"* to successfully compiled script. Scripts in the folders are like sandboxed apps allowing for failure and full recovery.
re : "I often see some divergent methodologies" - totally!
re : "probably difficult for them to see it "profitable" to just give it away for others to use freely" um, actually they made a few deals with the devil to not release what we need (investor/partners - soft confirmation from multiple sources). As for profitable, I will now quote Henke (~2+ drinks) on the Python scripts : "We would love to document the API, but then we would have to support it". Read between those lines as you wish...
re : "grateful that they haven't closed it completely" Yes, easy to do, but the genie is out of the bottle, updates would cease, pitchforks would be acquired!
re : Bitwig - personally - too much Java. If it was not for VBox I would be totally rid of that lang on my laptop. They used it for UI, yeah, I know the story - weak sauce. And for the record - Bitwig is funded by people who "hope to destroy Ableton". (I would love to credit that quote, sorry, can't do it). Same for Max - I don't like Java!
@thomasf you mention "catering to engineering needs of studio customization and/or artistic expressiveness" - wrong. Ableton is chasing bedroom producer / rookie money\ (that is the majority share of the market). I started a thread and hit 21 pages or more (I also hate forums, never visit that crap) highlighting these ideas and how they messed up for the pros on Live9. I have done demos and taught with most of the "favorite" Ableton craftsmen out there, some were cool/skilled, but most were clueless and trained as "Certified Sales people" offering liability and bad practice to the majority. Most bedroom producers have no frame of reference on music, software, production, or quality - so a cheapened product is an easy sell. Ableton is degrading itself to sell to the cattle leaving the pros behind. Ergo my absence. Otherwise you are correct on many points.
\ They are accountable to shareholders. Quality and direction are trumped by profit.
@rasputozen "I literally can't convince myself to reload live every time I tweak a script" The way the application bootstraps this is the only way it works***. Else you would need an API key or something to trigger an internal rebuild of Py scripts and that is a super HAWT attack vector for doing highly malicious activity. Think DDoS-ish the API, rainbow attack, various brute force, overflow/buffer roXXor. The reload hurdle keeps the script kitties at bay, generally.
*\ not totally sure, but I think the compiled objects are suspended in memory allocation lower in the application. So injecting that stuff into the UI+refresh+linkages would be a freaking nightmare. Us scripters in the API are of the slimmest minority - expect NO SUPPORT or consideration. We did a bunch of latency research on scripting styles and we considered much of the API scary solid and fast - I would love to know how they accomplished some of the benchmarks we saw messing around.
Don't take any of this crap out of context - I collaborated with Ableton and spent 4+ years punk'n them because they did not have their act together IMHO. Then they did me wrong several times, we had many yelling matches and fights over the same dumb stuff. As much as I was a punk, what I was doing underscored the chronic misunderstanding and cluelessness. I originally tried, in a friendly way, to help - but as I was told by many people "They live in fear of the legal team".
Ableton will succeed but their time is over. They will most likely be acquired in the years to come - it will be for the better. I put my $ on Yamaha as the new master.
thomasf
commented
9 years ago Ah, that covered the hate section well and full :) I'll add some hate of c++ for being too complete on too many levels and javascript for having insanely bad syntax so we have complete hate coverage for the programming languages used mostly by the sum of the products were discussing. Oh, and python for being an indentation significant language (in the way they chose to implement it).
rasputozen
commented
9 years ago Hey guys, I decided that it's still worth it to get at least some of these scripts figured out. I created a little ahk script to reload Live and delete .pyc files: https://github.com/rasputozen/live-reloader
It should be useful to anyone who's doing any kind of scripting in Live. If you have something better please let me know - or if you can make it better, please do! I also forked this repo and made some edits to it so you can clone it directly to your "MIDI Remote Scripts" folder in your Live install and keep all the original compiled Live scripts right alongside them: https://github.com/rasputozen/AbletonLive9_RemoteScripts
Feel free to contact me if you're interested in solving the Launchpad script - that's the one I'll probably be getting into in my spare time.
thomasf
commented
9 years ago I do not know if there are any relevant changes but you should maybe look at decompiling the 9.1.5 scripts if that is the version you are running before changing things.
rasputozen
commented
9 years ago In "About Live" my version just says "9.1 Build" so I'm not sure if it's 9.1.4 or 9.1.5 or what. How can I get the full version number?
rasputozen
commented
9 years ago Hey guys, definitely check out my new repos if you get a chance and let me know what you think/suggestions to make them better. I think my ahk script is pretty good and would help anyone scripting in live. But maybe you have something even better? Well clue me in! I want to be a better programmer!!
Also, do any of you have the original compiled launchpad.pyc script? When I first started editing the scripts I accidentally overwrote that specific file before I backed it up. If any of you could upload it somewhere or email it to me I'd really appreciate it.
rasputozen
commented
9 years ago Oh and Thom, I think the 9.1.4 scripts are probably fine. If we keep updating the scripts for every Live update we'll never get anywhere. This is just for me but I've turned off auto-update for this reason... what do the new Live updates really do anyway... I think I've been at 9.1.4 for probably a year now and haven't had any bugs that I can remember. Does Live release a change log? I'd be interested to see what's in that..
I think there's a stable torrent release of 9.1.4 as well so this increases the pool of scripters, which, I didn't do any studies or anything but I feel like people who torrent are more likely to be interested in reverse engineering the scripts. So I see holding off on updates until it's absolutely critical as increasing the pool of eligible scripters, which makes a bigger scripting community and hence more fun and progress.
thomasf
commented
9 years ago For one thing they just seems to have released 9.1.6 which finally supports high dpi resolutions ("retina") in osx, a big hooray for working on 13" macbook pro.
For windows I always keep a separate os install for A/V work (my studio os). I also never install any system updates at all unless I experience a specific problem. Having a stable experience with low latency audio interfaces in windows is one thing that has proven not so fun when blindly accepting any Microsoft updates over the years. This is of course the general problem with software which "has to" integrate by supporting common api versions and architectures.
Class compliant USB midi is awesome. A bunch of my older usb/pci midi/audio interfaces stopped working long before actually broke because there were no drivers anymore when I moved to 64bit windows. Nowerdays all new midi stuff even works on Linux straight out of the box and thats a big improvement for me.
I hope AVB or some other audio over Ethernet takes off as a standard for digital audio signals so we don't have to depend on the manufacturer updating drivers to continue using them.
Here are the release notes: https://www.ableton.com/en/articles/live-9-release-notes/ https://www.ableton.com/en/articles/live-9-beta-release-notes/
rasputozen
commented
9 years ago wow... I'm a total dope head... I just checked out Launchpad95 and found that it is actually a reverse-engineered version of the native launchpad scripts that are modified... So I don't need to reverse-engineer it myself since I suck at python. YAY!!
rasputozen
commented
9 years ago Hey Thom, thanks for all the help man. I forked the Launchpad95 script the other day and I was having issues with the user1 mode not working according to the docs. I saw the script was for 9.1.2 and I'm running 9.1.4. You think this could be caused by the framework scripts being updated since then? I saw you updated the script as well so I was wondering what version of live you're running? I opened an issue the but the project doesn't seem too active. Any insight you can offer? I really don't want to reinstall to 9.1.2.. but I may... I just may...
thomasf
commented
9 years ago It worked well at 9.1.4 and 9.1.5.. I have seen some bugs but I cannot remember which they were.. I almost exclusively use launchpad for clip launching only so I might not be someone who often spots things like that.
rasputozen
commented
9 years ago Ah I figured it out... shoulda tried a couple more things. I mistakenly thought the play was referring to arrangement view play. It seems to work fine. Thanks again for the help man, appreciate it.
gluon
commented
9 years ago I'm REALLY sorry for not taking part of your discussions here but I'm massively busy.
Enteleform
commented
2 years ago Welp... almost 10 years later.... Same problems :skull:
Soooo many restarts. RIP all of us :coffin:
*presses F to pay respects*
Anyone have any worthwhile updates to their personal workflows?
Has anyone managed to get IntelliSense for Live somehow? I'm about to start transforming structure-void.com/Live11.0.xml into Python typings, but if someone has a solution already that'd be extra dope, I'm kind of dreading it lol.
One thing I'm interested in but haven't tried yet is utilizing PyPI packages in scripts. Has anyone tried this?
@aumhaa
I read your "book" along with some other posts. They were very helpful and provided some valuable insight, thank you for being thorough!
I've tagged a couple of quotes from you in the sections below with some related updates/questions.
My personal setup uses a build system that generates a symlinked MIDI Remote Scripts directory which contains a mix of my own scripts, a small subset of the default Ableton scripts, and some 3rd-party scripts. This gives me a preferable method of organizing scripts than their default flat hierarchy, as well as a nice Git workflow.
Also have some VSCode/AHK/Python scripts to kill Live, load specific sets, wipe the log file, log my scripts' debugging output to separate files, etc.
And RE: IntelliSense, I have a mocked import system where my scripts import from @gluon's decompiled scripts so they get inferred typings, and then during the build step the import paths are adjusted to match what Ableton expects. This is accomplished via a directory convention that allows improved organization & directory-specific build artifacts.
They're also used to generate some mock MIDI scripts as an organization workaround @ Ableton's UI:
Just started a new single-script setup which I'm implementing as a modular system which can load specific modules per-project along with project-specific settings. I decided to try this out as a workaround to the 6-script limit (seriously... *wat-face-emoji*).
At this point I'm working on communicating with non-hardware interfaces via M4L devices and web apps, and will have a dedicated MIDI/OSC server to forward signals to an assortment of applications [Ableton, TouchDesigner, various web apps, etc.].
This is an example of a system I'll be integrating with:
[P5 Sketchbook] Preset Interpolation Demo
[P5 Sketchbook] Live Jam w/ MPK Mini MK3
Also have some additional vids about this system on my YouTube channel.
@aumhaa
You can't store settings along with Live sets.
I found a way that seems to be working pretty reliably so far, it generates a structure like this @ /<Project>/_Framework/:
where __Core__.json is generated if a track named [Framework] exists. __Core__ can then be updated manually updated to define which modules are enabled, and then only those modules are instantiated when the project is loaded. The module-specific JSON files are generated automatically the first time the system notices a particular module is enabled.
Here's a few files that demo the implementation details: https://gist.github.com/Enteleform/e2de9c8f4017dce3c26c019f523f5a57
The first Ableton-Specific module I'm working on is a hands-free pre-arranged looping workflow, inspired by zenAud.io ALK2.
I initially tried doing this in arrangement view to simulate the ALK2 workflow as closely as possible, via dedicated [Loop] control tracks which would manage the recording / playback of the track below.
I got the recording functionality working but then stopped before implementing the playback functionality because I ran into an issue where the recorded clips are several milliseconds off from the cue points, which would end up causing noticeable issues like dropped MIDI notes & audio pops. I tried two implementations for this; the Python API, and using MIDI loopback to use Ableton's native automation capabilities without triggering any scripts. Both implementations had similar latency.
@aumhaa
You can't schedule messages more often than 60ms
Have you managed to make any improvements in timing precision since your last posts?
I'd definitely prefer to go with the above implementation if it's possible. But the one below seems workable if precise arrangement timing still isn't possible, at least.
The implementation I'm working on now is kind of convoluted, but it seems to work! Basically, it still utilizes a pre-defined arrangement, but this time uses the timing data to generate a huge batch of ClyphX scripts to execute the arrangement in session view.
I drafted up the ClyphX scripts in the demo by hand (sooooo many... burned through hundreds while troubleshooting :skull:), but now that I know how they have to be shaped & implemented, the next step is to generate them programmatically.
This is a summary of the current functionality:
Playback mode to replay the recording without destroying the last performancePerformance mode clears the last performance and restarts the recording sequenceBTW if anyone knows of places where Ableton scripters congregate, please let me know!
I thought of posting this elsewhere and just dropping a link here but the only place I know of is the M4L forum, and this project doesn't really have much M4L-relevance just yet. Oh and the Ableton forum, but that seems fairly sparse on these topics.
rasputozen
commented
2 years ago This was kind of a trip to read through again. 8 years lol! Feels like literally a different lifetime lol. I ended up dropping Windows and Live for 6 years and worked exclusively on a linux box. I got a lot better at programming but my mind hasn't been directed toward DAWs or production since lol! (also linux sucks in its own way and I'm currently back with Win on a SurfacePro b/c it was the best deal for a fanless laptop)
And it seems like Live hasn't changed much xD
I want to start fixing the bugs in the launchpad scripts from incorrect decompilation of the .pyc files. Is this the right repo to start making pull requests as I make progress? Or is that not the goal of this project and I should look elsewhere or fork my own?
I'm pretty new to git, and I was wondering why this project isn't called "MIDI Remote Scripts"? I made my local repository the actual "MIDI Remote Scripts" folder in the User Library so I could easily test my code in ableton as I'm working on it. When I cloned this repo, however, it made a folder with the title of this project. So I copied all the script directories out of this folder and pasted them in the "MIDI Remote Scripts" folder. Will this affect pull requests?
I've tried to get ableton to recompile the .py files I edit by selecting "none" in the control surface drop down menu and then going back to the desired script. This doesn't cause ableton to recompile the new code though... so I have to close out of ableton and open it back up in order to test the code... very slow and inefficient. Is there a way to keep ableton open and test new code as I work on the scripts?