XMI File Support - Githubissues

psi29a commented 10 years ago

We have two choices here:

Use Chris's beta patch and work on that until it works
Use a C version of Thirdeye's implimentation

Either way, we should validate against Thirdeye's music reproduction.

psi29a commented 10 years ago

https://sourceforge.net/p/wildmidi/feature-requests/5/

From Chris:

I am a little cautious about adding support for this format as it is a commercial format solely for the Miles Sound System, and the only other references for it is in a commercial products. So far I have not located an open description of the file format or standard, nor a .xmi file i can freely download, but my search will continue.

Can you please provide further open documentation for the "Miles Sound System extended MIDI" format, and links to .xmi files that can be freely downloaded for non-commercial use.

Addition info: http://forum.zdoom.org/viewtopic.php?f=15&t=26046 http://fheroes2.svn.sourceforge.net/viewvc/fheroes2/trunk/fheroes2/src/engine/

SandySandy (afletdinov) Patch: https://dl.dropboxusercontent.com/u/396161/initial_xmi_support_v2.diff

sezero commented 10 years ago

Another pointer for documentation: http://fabiensanglard.net/reverse_engineering_strike_commander/docs/XMIDI.TXT

AIL2.ZIP from Miles' site @ http://www.ke5fx.com/ has the assembly source (http://www.thegleam.com/ke5fx/misc/AIL2.ZIP). IANAL, but the package says "As of May 26, 2000, all source code and compiled object code in this package may be considered "open-source freeware," usable by anyone for any purpose, commercial or otherwise, without restriction or limitation."

On 2/25/14, Bret Curtis notifications@github.com wrote:

https://sourceforge.net/p/wildmidi/feature-requests/5/

From Chris:

I am a little cautious about adding support for this format as it is a commercial format solely for the Miles Sound System, and the only other references for it is in a commercial products. So far I have not located an open description of the file format or standard, nor a .xmi file i can freely download, but my search will continue.

Can you please provide further open documentation for the "Miles Sound System extended MIDI" format, and links to .xmi files that can be freely downloaded for non-commercial use.

Addition info: http://forum.zdoom.org/viewtopic.php?f=15&t=26046 http://fheroes2.svn.sourceforge.net/viewvc/fheroes2/trunk/fheroes2/src/engine/

SandySandy (afletdinov) Patch: https://dl.dropboxusercontent.com/u/396161/initial_xmi_support_v2.diff

Reply to this email directly or view it on GitHub: https://github.com/Mindwerks/wildmidi/issues/7#issuecomment-36017676

psi29a commented 10 years ago

I've had the pleasure of talking with John and he considered it all "Public Domain". I also implemented my own XMIDI (XMI) support (with help form Ryan Nunn of Exult) here: https://github.com/psi29a/thirdeye/tree/master/apps/thirdeye/sound

It is more accurate than existing xmi2midi out there.

psi29a commented 10 years ago

In the xmi branch is a work-in-progress for xmi support. It does work, but beware of dead code and other things that need to be taken care of. If problems are found, please post them here.

psi29a commented 10 years ago

For those wishing to test, here are some common XMI files: http://www.vgmpf.com/Wiki/index.php?title=Category:Games_That_Use_XMI

Current status: It works, but there are still some clang and valgrind issues.

Once these are solved, this branch will be merged in as our first new feature of the 0.4 series.

psi29a commented 10 years ago

Clang issues are solved, now just some memory leaks.

sezero commented 10 years ago

Multri-track xmi files, e.g. those from Legend of Kyrandia, don't play because we set midi format to 0 and only retrieve track/0. Needs addressing.

psi29a commented 10 years ago

Yup, it is also on the list... we should be able to play all tracks in an XMI.

chrisisonwildcode commented 10 years ago

Just want to point out that wildmidi supports type 0 (all channels on one track) and type 1 (1 channel per track) midi file format. To support type 2 (each track is a separate type 0) would not be hard to support but finding a type 2 to test with could be a challenge without manually constructing one.

Chris Ison

Sent from my iPhone

On 29 Mar 2014, at 2:01 am, Bret Curtis notifications@github.com wrote:

Yup, it also on the list... we should be able to play all tracks in an XMI.

Reply to this email directly or view it on GitHubhttps://github.com/Mindwerks/wildmidi/issues/7#issuecomment-38935659 .

sezero commented 10 years ago

Just want to point out that wildmidi supports type 0 (all channels on one track) and type 1 (1 channel per track) midi file format. To support type 2 (each track is a separate type 0) would not be hard to support but finding a type 2 to test with could be a challenge without manually constructing one.

Type 2 support would be cool, although I don't know how I would find sample midi files to test with. Looking at libtimidity, it seems to append newer tracks to previous ones.

BTW, multi-track xmidi files from Legend of Kyrandia don't play correctly if I change xmidi.c to export a type/1 midi (I might have made mistakes in my code though..), so I assume that you already knew that we'd need type/2 support in order to support multi-track xmi files?

sezero commented 10 years ago

Applying the following rough patch to the xmi branch outputs a type/2 out.mid file which libtimidity can play. I don't know about any subtleties that needs to be taken care of, though.

diff --git a/src/xmidi.c b/src/xmidi.c index b4f9914..c810d54 100644 --- a/src/xmidi.c +++ b/src/xmidi.c @@ -538,12 +538,13 @@ uint32_t xmi2midi(struct xmi_ctx *ctx,


    write4(ctx, 6);

write2(ctx, 0);
write2(ctx, 1);
write2(ctx, (ctx->info.tracks > 1)? 2 : 0);

write2(ctx, ctx->info.tracks);
write2(ctx, ctx->timing[track]);
}

len = ConvertListToMTrk(ctx, ctx->events[track]);
for (i = 0; i < ctx->info.tracks; i++)

len += ConvertListToMTrk(ctx, ctx->events[i]);
/* cleanup /
if (ctx->events) {
@@ -558,6 +559,14 @@ uint32_t xmi2midi(struct xmi_ctx ctx,
ctx->timing = NULL;
ctx->fixed = NULL;


+#if 1

if (ctx->dst) {
FILE *out = fopen("out.mid","wb");
fwrite(ctx->dst,ctx->dstsize,1,out);
fclose(out);
}
+#endif
+
return (14 + len);
}

sezero commented 10 years ago

The xmi support is now merged into the master branch and further development should me done therein.

TODO:

The multi-track xmidis from Legend of Kyrandia really seem to need type-2 midi support which should be added to wildmidi_libc.c after which xmidi.c can be adjusted like described at https://github.com/Mindwerks/wildmidi/issues/7#issuecomment-39006134
The two-pass conversion of initXMI()+xmi2midi() should be changed into a one-pass xmi2midi()
The number of malloc() calls during events list preparation can be reduced
(Anything else?)

sezero commented 10 years ago

Applied 8e22d91, which :

eliminates 2-pass conversion, does the job in 1-pass using realloc
eliminates initXMI and does only xmi2midi by combining old initXMI+ old xmi2midi.
doing the above removes requesting a specific track. (see below.)
adds support for multi-track xmi files by outputting type-2 (which needs implementing in wildmidi_lib.c properly) instead of type-0: if type-2 is implemented, then specific track requests can be done when calling wildmidi_lib functions instead of xmidi converters.
removes 'fixing of broken midis' (ctx->fixed and its associated MovePatchVolAndPan procedure.)

We need to implement type/2 midi support.

psi29a commented 10 years ago

This issue is more or less resolved right? Type-2 midi support is another issue, once that is finished then we can test LoK xmidis.

sezero commented 10 years ago

On 3/31/14, Bret Curtis notifications@github.com wrote:

This issue is more or less resolved right?

Yes

Type-2 midi support is another issue, once that is finished then we can test LoK xmidis.

LoK midis, also DEMO.XMI from Miles' AIL2.ZIP, and who knows how many more.

BTW, if you enable the out.mid dumping code in xmidi.c by changing if 0 to if 1, you can test the resulting type/2 out.mid file using any midi player: libtimidity supports type/2 by simply playing the tracks sequentially, try my 'wildtimi' (wildmidi player+libtimidity backend) from uhexen2.sf.net/tmp/ for example.

sezero commented 10 years ago

On 3/31/14, Bret Curtis notifications@github.com wrote: This issue is more or less resolved right?

Yes

Closing this now.

chrisisonwildcode commented 10 years ago

Reopening this temporarily because I need to know if wildmidi-devtest has any issues with any XMI's you have. Please see the DevTest-Updates branch of my fork https://github.com/chrisisonwildcode/wildmidi/tree/DevTest-Updates

sezero commented 10 years ago

On 5/5/14, chrisisonwildcode notifications@github.com wrote:

Reopening this temporarily because I need to know if wildmidi-devtest has any issues with any XMI's you have. Please see the DevTest-Updates branch of my fork https://github.com/chrisisonwildcode/wildmidi/tree/DevTest-Updates

Not actually a DevTest result, but xmi conversion abuses the heap a lot, (I believe Bret had noticed and reported this too, can't remember where), whereas mus conversion is a lot more a good boy. xmi2midi's eventlist generation is the culprit.

chrisisonwildcode commented 10 years ago

Not touching xmi2midi, only referring to wildmidi-devtest binary ... especially if you get a failed result

sezero commented 10 years ago

On 5/5/14, chrisisonwildcode notifications@github.com wrote:

Reopening this temporarily because I need to know if wildmidi-devtest has any issues with any XMI's you have. Please see the DevTest-Updates branch of my fork https://github.com/chrisisonwildcode/wildmidi/tree/DevTest-Updates

Needed the following obvious patch to compile

DevTest.c:656: error: ‘for’ loop initial declaration used outside C99 mode


diff --git a/src/DevTest.c b/src/DevTest.c
index f07e1a9..045de39 100644
--- a/src/DevTest.c
+++ b/src/DevTest.c
@@ -524,7 +524,7 @@ int check_midi_event (unsigned char midi_data,
unsigned long int midi_size,
int test_xmidi(unsigned char \ xmidi_data, unsigned long int xmidi_size,
int verbose) {
unsigned int tmp_val = 0;


unsigned int i = 0;
unsigned int i = 0, j;
unsigned int form_cnt = 0;
unsigned int cat_len = 0;
unsigned int subform_len = 0;
@@ -653,7 +653,7 @@ int test_xmidi(unsigned char * xmidi_data,
unsigned long int xmidi_size,
patch information
*/
tmp_val /= 2;
for (unsigned int j=0; j < tmp_val; j++) {

for (j = 0; j < tmp_val; j++) { if (verbose) printf ("Patch:%i, Bank:%i\n", xmidi_data[0], xmidi_data[1]); xmidi_data += 2;

Almost all of the xmidis from Eye of the Beholder, Legend of Kyrandia, SimCity2000, Ultima Underworld and System Shock I threw at it reported 'Success', except for INTROTHM.XMI (both 'GENMIDI' and the 'SBLASTER' versions of them) and TITLOOP.XMI (only the 'GENMIDI' version) from System Shock, and UW11.xmi and UW13.xmi from Ultima Underworld. However, they do play without error in wildmidi player.

Sent the files and the devtest debug outputs of them to your personal email.

chrisisonwildcode commented 10 years ago

@sezero fixed that error :)

chrisisonwildcode commented 10 years ago

@sezero I'll work on the "failed" XMI issue today some time. This is the result of a chunk of XMI data not being documented well in the info I had and I had no XMI that had that chunk. I will add support for that chunk and update here when I've committed it.

chrisisonwildcode commented 10 years ago

@sezero Those files should pass DevTest now :) I still have working out to do with the "RBRN" chunk but they won't fail just for having that chunk.

sezero commented 10 years ago

On 5/7/14, chrisisonwildcode notifications@github.com wrote:

@sezero Those files should pass DevTest now :) I still have working out to do with the "RBRN" chunk but they won't fail just for having that chunk.

About the RBRN entry: AIL2.ZIP from Miles' site @ http://www.ke5fx.com/ http://www.thegleam.com/ke5fx/misc/AIL2.ZIP A214_D3.ZIP contained in that AIL2.ZIP has a MIDIFORM.C that may help with those branch points. Also this one has some documentation: http://fabiensanglard.net/reverse_engineering_strike_commander/docs/XMIDI.TXT

psi29a commented 10 years ago

When we're finished with formats, we should add documentation here: https://github.com/Mindwerks/wildmidi/tree/master/docs/formats

We should probably take that last bit, improve it a bit and commit it.

chrisisonwildcode commented 10 years ago

Can someone please confirm that RBRN chunk is ignored for XMIDI to MIDI conversion as it appears to be ignored in our xmidi.c (do not see a reference to it in the source)

Sent from my iPhone

On 7 May 2014, at 8:36 am, sezero notifications@github.com wrote:

On 5/7/14, chrisisonwildcode notifications@github.com wrote:

@sezero Those files should pass DevTest now :) I still have working out to do with the "RBRN" chunk but they won't fail just for having that chunk.

About the RBRN entry: AIL2.ZIP from Miles' site @ http://www.ke5fx.com/ http://www.thegleam.com/ke5fx/misc/AIL2.ZIP A214_D3.ZIP contained in that AIL2.ZIP has a MIDIFORM.C that may help with those branch points. Also this one has some documentation: http://fabiensanglard.net/reverse_engineering_strike_commander/docs/XMIDI.TXT

— Reply to this email directly or view it on GitHubhttps://github.com/Mindwerks/wildmidi/issues/7#issuecomment-42368687 .

psi29a commented 10 years ago

Do we have a file we can test with RBRN in it? It is considered optional.

chrisisonwildcode commented 10 years ago

We do, I am just wondering if you guys aware of the RBRN chunk being used in any 2midi program for something within the midi ...

Sent from my iPhone

On 7 May 2014, at 7:24 pm, Bret Curtis notifications@github.com wrote:

Do we have a file we can test with RBRN in it? It is considered optional.

— Reply to this email directly or view it on GitHubhttps://github.com/Mindwerks/wildmidi/issues/7#issuecomment-42406144 .

psi29a commented 10 years ago

Do you mind sharing this file? :) (That is what I was asking.)

sezero commented 10 years ago

On 5/7/14, Bret Curtis notifications@github.com wrote:

Do you mind sharing this file? :) (That is what I was asking.)

Two of them (ones from ultima underworld) are actually available vgmpf.com but just sent to your personal email anyway.

psi29a commented 10 years ago

Thanks! :)

Don't have access to a test environment at the moment, can you answer Chris' question? Do we just ignore this chunk or is it being processed?

chrisisonwildcode commented 10 years ago

Can you please rename XMIFileFormat.txt to IBM_XMIFileFormat.txt?

Want to write our own version XmiFileFormat.txt in a similar style of the 2 I have already done.

psi29a commented 10 years ago

Done

sezero commented 10 years ago

On 5/9/14, chrisisonwildcode notifications@github.com wrote:

Can you please rename XMIFileFormat.txt to IBM_XMIFileFormat.txt?

To my knowledge, this has nothing to do with IBM except for it is actually part of IBM PC Audio Interface Library, i.e. AIL of Miles, so the name doesn't make any sense + wrong.

chrisisonwildcode commented 10 years ago

The file I was referring to is an IBM file from their AIL docs :)

Sent from my iPhone

On 10 May 2014, at 12:34 am, sezero notifications@github.com wrote:

On 5/9/14, chrisisonwildcode notifications@github.com wrote:

Can you please rename XMIFileFormat.txt to IBM_XMIFileFormat.txt?

To my knowledge, this has nothing to do with IBM except for it is actually part of IBM PC Audio Interface Library, i.e. AIL of Miles, so the name doesn't make any sense + wrong.

— Reply to this email directly or view it on GitHubhttps://github.com/Mindwerks/wildmidi/issues/7#issuecomment-42672631 .

chrisisonwildcode commented 9 years ago

You will be happy to know that this is well under way and I should have something uploaded tonight (my time) for testing provided I have not majorly screwed up some where.

psi29a commented 9 years ago

That is awesome news! :)

Just so you know, you don't have to implement every new (old?) format. We just want to have the major bugs you found (linear volume?) squashed and the MUS/XMI/HMP/HMI (if they are finished, or good enough for now) formats in your branch so we can merge it in for a proper 0.4.0 release. Sezero and I can also work on more features/formats using yours as an example to follow.

chrisisonwildcode commented 9 years ago

With the latest commit xmi.c is "good enough".

NOTE: if any issues with playback speed come up, please provide a recording of the audio from the game of the xmi playing so that a proper comparison can be done.

NOTE2: xmi.c treats the midi data contained within the XMI file as GM. Will add adherence to conversion options at another time.

NOTE3: Noticed that xmidi.c only uses the first tempo change and ignores all others. Issue?

psi29a commented 9 years ago

Need to think about this...

psi29a commented 9 years ago

So we have xmi support, but it could be fixed up with extra features. I'll break this ticket off into sub-tickets for 0.4.1 (or later).

What do you guys think?

chrisisonwildcode commented 9 years ago

Yes, go for it .. break it off into a TODO for 0.4.1 ... i.e.: xmi playback does not adhere to conversion options.

psi29a commented 9 years ago

Not sure if relevant, but this xmi sounds off on your branch in comparison to master. The result of your branch yields output that is slow... (double the length).

I'll email it to everyone. Don't have dropbox handy at the moment.

psi29a commented 9 years ago

Email sent.

chrisisonwildcode commented 9 years ago

I don't think I received the email from you ...

chrisisonwildcode commented 9 years ago

Ok. looks like xmi does only look at first tempo ... the 2nd tempo in that xmi is slower than the first one ... and slower than the theme I heard on youtube from a gameplay video.

chrisisonwildcode commented 9 years ago

try it now

psi29a commented 9 years ago

master branch:

Playing 1.FSEQ 
q       [Approx  1m 59s Total]

latest wildcode-updates:

Playing 1.FSEQ 
q       [Approx  2m 36s Total]

still doesn't sound quite right...

sending new dump via email

chrisisonwildcode commented 9 years ago

Please note that master branch has the bug that ignored some samples until next midi event making the play time shorter than it should of been. Both midi files are the same length (in time) in Logic Pro 9 & X

chrisisonwildcode commented 9 years ago

I should also add that 0.3 had the same bug

psi29a commented 9 years ago

Ah! Great catch, that. :) I think the sound is off because you currently don't do any conversions while master did mt32 to GM. I'll test tomorrow morning and if it tests out alright, I'll close this.

Mindwerks / wildmidi

XMI File Support #7