C64 SID music import

[doyousketch2]

LMMS already has a SID effect generator, I think it would be awesome if we could import SID files directly, and then we'd have access to a vast treasure trove of mixable material via hvsc.c64.org

I posted this question on Facebook, and was invited to add this request to your GitHub tracker.

While I do some programming, I think this may be a bit beyond my ability. I'll do my best to bring relevant info on the subject to aid in making it happen.

When you download the HVSC, it contains a text describing the SID file format in the documents directory. Here's a link so it's easier for you to view: http://www.hvsc.de/download/C64Music/DOCUMENTS/SID_file_format.txt

Under the Players and Editors topic you can see a few songs disassembled: http://codebase64.org/doku.php?id=base:sid_programming

Some PDF's that show the inner workings of C64 music files http://digilander.libero.it/ice00/tsid/sidin/

This helped explain the "4th channel" trick http://www.pauliehughes.com/page22/page22.html

If you ramp up the volume all the way on the old 6581 you'd get a click.

They'd basically pulse-width modulate how fast it would click full-on, so you'd get an extra sound channel.

Sorta like people do when they play music on a floppy disk or printer.

These links explain the differences of Compute's Gazette alternative .MUS file http://sidplayer.org/filestru2.txt http://www.unusedino.de/ec64/technical/formats/sidplay.html

I'm guessing it would go something like this:

Load File > File Parser

Song and composer info could be passed to the Project notes.

Set up the Sid plug in as an instrument.

Scan through once to link all the necessary automation tracks to their respective controllers.

Set up a timer, 50hz for PAL or 60hz for NTSC regions

Start pulling in note data, plus fill in the automation tracks where needed.

Follow the patterns until it's finished

Then we can tweak out all those wonderful chiptunes of yesteryear to our heart's content.

[pbella]

Wouldn't this need be better served by an audio file conversion utility? (i.e. convert the SID to MIDI or WAV file which LMMS can work with)

I can (sort of) see the appeal of the SID synth for music/sfx, but to start dealing with converting obsolete file formats doesn't seem like the best use of resources.

[softrabbit]

Set up the Sid plug in as an instrument.

Actually 3 of them. The SID plugin in LMMS has all oscillators in use for each note and can't play 3 independent notes.

Scan through once to link all the necessary automation tracks to their respective controllers. Set up a timer, 50hz for PAL or 60hz for NTSC regions Start pulling in note data, plus fill in the automation tracks where needed. Follow the patterns until it's finished

More like "execute 6502 code and record writes to the SID chip as notes and automation in LMMS". Those songs are basically C64 programs, not readable data structures.

[unfa]

Sounds like a way to get new crazy people into LMMS community :D

The major question is how hard it will be to translate .SID instruments to make them sound as close to the original as possible.

What about dual-chip compositions? Some folks plugged another SID chip into their Commodore, to make more complex music. 30 sty 2015 03:33 "doyousketch2" notifications@github.com napisał(a):

LMMS already has a SID effect generator, I think it would be awesome if we could import SID files directly, and then we'd have access to a vast treasure trove of mixable material via hvsc.c64.org

I posted this question on Facebook, and was invited to add this request to your GitHub tracker.

While I do some programming, I think this may be a bit beyond my ability. I'll do my best to bring relevant info on the subject to aid in making it happen.

When you download the HVSC, it contains a text describing the SID file format in the documents directory. Here's a link so it's easier for you to view: http://www.hvsc.de/download/C64Music/DOCUMENTS/SID_file_format.txt

Under the Players and Editors topic you can see a few songs disassembled: http://codebase64.org/doku.php?id=base:sid_programming

Some PDF's that show the inner workings of C64 music files http://digilander.libero.it/ice00/tsid/sidin/

This helped explain the "4th channel" trick http://www.pauliehughes.com/page22/page22.html

If you ramp up the volume all the way on the old 6581 you'd get a click.

They'd basically pulse-width modulate how fast it would click full-on, so you'd get an extra sound channel.

Sorta like people do when they play music on a floppy disk or printer.

These links explain the differences of Compute's Gazette alternative .MUS file http://sidplayer.org/filestru2.txt http://www.unusedino.de/ec64/technical/formats/sidplay.html I'm guessing it would go something like this:

Load File > File Parser

Song and composer info could be passed to the Project notes.

Set up the Sid plug in as an instrument.

Scan through once to link all the necessary automation tracks to their respective controllers.

Set up a timer, 50hz for PAL or 60hz for NTSC regions

Start pulling in note data, plus fill in the automation tracks where needed.

Follow the patterns until it's finished

Then we can tweak out all those wonderful chiptunes of yesteryear to our heart's content.

— Reply to this email directly or view it on GitHub https://github.com/LMMS/lmms/issues/1722.

[badosu]

Well, I am not against this. But code that translates sid -> midi should be an executable apart from lmms. Then we can think of integrating it.

[doyousketch2]

Wouldn't this need be better served by an audio file conversion utility? (i.e. convert the SID to MIDI or WAV file which LMMS can work with)

MIDI would get the note data, however it would leave out the ASDR envelope and just pick a generic instrument to fit in. WAV would be OK for some purposes, but you can't easily pull one instrument from a rendered polyphonic recording.

Set up the Sid plug in as an instrument. Actually 3 of them. The SID plugin in LMMS has all oscillators in use for each note and can't play 3 independent notes.

Yeah, you're right. That's what I was originally thinking, but I second-guessed myself 'cuz I was tired. In that case, we'd have to give the Ring-modulators the ability to run global so they will affect the other SID generators. In the right-click menu for Ring there could be a toggle for local or global effect, and it would default to global while importing, otherwise local so it just affects the one SID instance.

More like "execute 6502 code and record writes to the SID chip as notes and automation in LMMS".

Yeah, that's what I meant. I assumed anyone who knew what we were talking about would get that, but it's 'prolly best to spell it out so we're all on the same page.

Those songs are basically C64 programs, not readable data structures.

That's the great part. They are essentially already just like LMMS music pattern blocks with automation for the SID parameters. The readability is only goverened by what we do with it. I'm looking at the source code of libsidplayfp (which was just updated a few days ago so it's current) to see if we can glean the appropriate data with those routines. http://sourceforge.net/projects/sidplay-residfp

Sounds like a way to get new crazy people into LMMS community :D

Yeah, I just messaged Seth Sternberger of 8bit Weapon. He gave me a thumbs up on the idea already. https://www.facebook.com/8bitweapon/posts/1025270604153172

The major question is how hard it will be to translate .SID instruments to make them sound as close to the original as possible.

The code that LMMS uses for it's SID emulation appears to be based off of ReSID, and, according to wiki, that stopped development in 2004 - https://en.wikipedia.org/wiki/ReSID - so it's currently as accurate as anyone had made it up to that point. It can be enhanced by bringing in the libsidplayfp code, as it's more up-to-date and the sound accuracy has been improved upon in these 10 years or so.

What about dual-chip compositions? Some folks plugged another SID chip into their Commodore, to make more complex music.

That should be no problem for LMMS, as you'd just put in the required number of SID plugins to cover the amount of voices used. So, as softrabbit pointed out, it would be 3 for a standard SID file, and thus, 6 for stereo. You might end up with a lotta automation tracks if the piece of music is quite involed.

Well, I am not against this. But code that translates sid -> midi should be an executable apart from lmms. Then we can think of integrating it.

As I'd mentioned a moment ago, MIDI is not ideal. There is a Windows program that can export all the required info to a text file, called SID2MIDIw - https://sites.google.com/site/joachimwijnhoven/downloads - and while it's neat, it sorta leaves out Linux and Mac users, so that's not ideal either.

I'd like to put in examples, but I haven't slept. My kid has a birthday party to go to, so I'll see if I can get a few examples in while he's out. But yeah, I gotta crash for now.

[doyousketch2]

OK, so I'm not averse to a separate way to import them, and I'm conjuring up a method to do so. It may be way easier than trying to tweak the LMMS source, but I need to know more details about the way LMMS saves files.

I'm looking at - lmms/src/core/DataFile.cpp - and what I'm noticing is that .MMPZ appears to be, at it's core, an .XML file that's got all the appropriate attributes in the proper places.

The thing is, it appears to be compressed. I tried renaming the file .ZIP .7z .BZ .LZ .LZH and .GZ with the assumption that the Z at the end indicates that it's using one of those compression algorithms.

.GZ is sorta open-able in Squeeze, it'll show that something's contained within, but it won't extract to a human-readable format. Xarchiver and 7-Zip don't seem to do anything with it. Am I missing something?

Does anyone know if the contents are extractable? What format are .MMPZ files compressed with?

My hope is to convert the SID2MIDIw - text output into XML that's readable by LMMS. I think that's do-able, 'cuz all the info should be there, I just need to know how to format it. So if I had the above question answered, it would go a great way to solving this.

[someguy1233]

Mmm, seems like at least version 0.4.14-rc1 stores data in a compressed or uncompressed form of QtXml. It needs to be parsed with some QtXML parser.

See the archive at the sources. src/core/mmp.cpp Specially the method that starts with

void multimediaProject::loadData( const QByteArray & _data, const QString & _sourceFile ) { ---- some code here .... }

it calls http://doc.qt.io/qt-5/qbytearray.html#qUncompress to load a QByteArray thats contains everything, then it pass to setContent(uncompressed, ..) if useful. I don't have so much time to analyze setContent(..), but maybe, that is the path to your answer.

And, the file format is version dependant. If you see "void multimediaProject::upgrade()" method, you will see what i'm talking about.

Maybe someone can write a script for a conversion "out of the box" meanwhile.

Edit update: Actually I figured out that multimediaProject is an extension class of QDomDocument that's part of qtXML, so there is no documentation of how data is grouped inside it. The code seems to be likely querying for key/path inside a root filesystem, and I have not found already the piece of code that queries for midi and automation data during playback. How to link the automation to controllers, is not easy to figure out without some standarization and documentation. Yo will need to read the code.

[doyousketch2]

Thanks for looking. Qt is a cross-platform GUI library, so I wouldn't think that's it, but maybe it has a built in compression routine. I'll dig deeper. TAR.BZ TAR.BZ2 and TAR.GZ were a no-go. I did see something about Base64 referenced in the LMMS source code somewhere, so I was wondering if that had something to do with it.

[someguy1233]

Yes, It seems to be a zlib compatible compression algorithm and maybe the encoding is base64, but the structure of XML and the data model is not defined explicitly in a form that we can read without reading and analyzing the code.

Also, inside this XML, XMMS stores all project information, not only the MIDI and automation for only one instrument. There is a lot of data and metadata. So, to figure out where data (the real thing that we want to import) is written inside a .mmpz (again a zlib base64 encoded XML that contains a freezed qDomDocument object), first we need to have some information on how the instrument track is loaded, and how it is played inside the code. Somewhere inside lmms, this qDomDocument object is queried for a midi (or midi like data) track, stored in some path inside it. I'm new to lmms code, but I have experience in c++. There is a new jungle to me, so maybe can help.

Edit: The code have improved a lot from 0.4.14 to 1.x.x version, so I'll need to read the new version. The loading process is somewhere rewritten, but in the basics is essentially the same.

[doyousketch2]

OK, it is, as I suspected, a compressed XML. I dunno why it wasn't unzipping. A while back my kid wanted me to install Steam, and I think that botched a lot of stuff. WINE no longer functions. I'll 'prolly wipe the drive and start with a fresh copy of SolydX when I get a chance.

Anyway, I found this link - http://www.linuxmusicians.com/viewtopic.php?p=11590 that suggests you can "upgrade" the .MMPZ into an .MMP

lmms --upgrade myFile.mmpz myFile.mmp

I'll tinker around with putting in values from SID2MIDIw text-output tomorrow. I'm happy we're getting somewhere. This seems promising. I guess I'm just wondering if we can simply zip an .MMP and it'll behave, or if specific flags have to be set.

If anyone knows, that's cool. If not, I'll tinker around with it for a couple days and see what I can make of it. We're supposed to be snowed in like a foot or so, so this'll gimme something to do.

[Spekular]

@doyousketch2 It does use Qt for compression, iirc it's called QCompress. You can easily get an .mmp by clicking save as in LMMS and removing the final z in the extension, it'll save properly (uncompressed). You can load .mmp s too so that's not a problem.

[someguy1233]

Great. I read the plain XML now. The midi information seems to be stored on <pattern></pattern> Note volume, timing and duration of a single note seems to be stored on:

<note pan="value" key="value" vol="volume" pos="position" len="length">

inside <pattern> instrument is defined in <instrumenttrack>

Maybe the only thing to do is to start an empty project with only a sid instrument and then import the midi from project dialog. Is that possible? If that is possible, half of the work is done, next you need to figure out how to reconstruct the automation parameters of the sid controller from the sid file.

[musikBear]

with only a sid instrument and then import the midi from project dialog. Is that possible?

Not natively, midi-import will apear inside a sound-font-track, and lmms will use the soundfont, whitch is specified in settings as 'default-soundfont' But what will happend, if you actually replace that with a sid.. ! -i doubt it will work, though

[someguy1233]

If you ignore the error of soundfont, and then copy the midi track inside the sid, and then remove the sf2 it will work. Main tempo maybe lost, you will need to adjust manually.

The problem with sid files is that they are most likely machine code, to be executed on a c64 like machine environment (virtual or not). These files not only have midi information, also have low level machine instruction used to tweak the sid chip during execution to sound differently as it was designed. It will be hard to reproduce only with midi information, oscillator configuration and other automation parameters will be hidden. Also if we have all of that information, maybe the whole thing will not sound exactly equal because timing definition of automation is not precise enough. And maybe there is no direct access to the chip registers from automation (the last two things needs to be checked).

But if it works, it would be great, because we can see them, in lmms, and tweak them.

[doyousketch2]

Think I'm getting somewhere.

Here's an LMMS save file with the first few measures of Matt Gray's "Driller" http://wikisend.com/download/182474/Driller-Intro.mmp

This was the text file I used to create it http://wikisend.com/download/390140/Driller Intro.txt

If you want to hear the original, try this link: http://www.wothke.ch/tinyrsid/index.php/explorer then click Musicians > G > Gray_Matt > Driller.sid

There's a column labeled Freq in the text output, and I'm not sure what it does exactly. I was trying to figure it out from: Games > M-R > Paradroid.sid so I think the C64 isn't limited to 2 octaves of course detuning.

I'm guessing it had unlimited portamento. Once you start a note, you can bend it to any other note you want.

http://dvfugit.com/beats-per-minute-millisecond-delay-calculator.php

I settled on 750 bpm 'cuz it syncs to the Vertical blanking interval NTSC is 60 hz or 0.02 ms or 750 bpm

I settled on those numbers because I quantized the automation tracks to a 16/th note. This could be reduced to 375 bpm if automation is quantized 1/32th... or 187.5 bpm at 1/64th

[Sti2nd]

Sounds awesome dude! :blush:

I think 750 bpm is a bit high, so yeah, larger quantization value in automation editor could be the way to go.

so I think the C64 isn't limited to 2 octaves of course detuning

One could probably increase that :)

[doyousketch2]

Gimme a couple days to play with more SID files before we change LMMS.

I'm programming a way to transcribe the flat text into XML. http://scratch.mit.edu/projects/46518132

After I have the script built up, I'll see about converting it to Python http://wiki.scratch.mit.edu/wiki/Kurt

It should convert faster that way.

If all goes well, we can play a few in LMMS and get a better idea of what we can do with 'em.

[Sti2nd]

Sounds like a plan!

[unfa]

Looking forward to see this in action :) 5 lut 2015 10:05 "Stian Jørgensrud" notifications@github.com napisał(a):

Sounds like a plan!

— Reply to this email directly or view it on GitHub https://github.com/LMMS/lmms/issues/1722#issuecomment-73014693.