Bug: invalid Wave: truncated

[X-Raym]

Hi !

I have some file which display this in the console: invalid Wave: truncated

The metadata displays just fine in other application I use, like REAPER.

If other apps can display this right, maybe BWFMetaEdit is a bit too strict ? Or, how could this be fixed right on the file ?

Thanks for your support !

[dericed]

Letting you know that your wav file is malformed isn't simply being strict but being informative. Although other applications don't have these warnings in scope, this type of validation issue is certainly in scope of BWF MetaEdit. Your wav file likely has a size declaration in its header that is larger than the actual file, so you file is likely a partial or truncated copy of another file and there is audio at the end that is missing.

[X-Raym]

@dericed Oh yes it is informative, but not having, the possibility to see/process information, is restrictive (considering that other apps can show them). It is not just a informative warning, I can't for eg dump the avaible metadata.

Apart from re-render the file, is there anyway this could be fixed ?

[JeromeMartinez]

the possibility to see/process information, is restrictive

With other tools you thought that all is OK, this is not the case. It is already a good thing to have a messenger saying something went wrong, don't you think? Then you can use other tools for digging in the file.

I can't for eg dump the avaible metadata.

Which "metadata" do you need? the exact position and value of the issue? True, it is not in BWF MetaEdit because it was not expected in the design, when we started the project. It could be implemented if there is enough need about it. If you look for a byte per byte dump of the file right now, you can check MediaConch, open the file for checking then click on "MediaTrace" button for the file. You still need some knowledge about WAV structure as the exact issue is not showed (it is only a dump).

Apart from re-render the file, is there anyway this could be fixed ?

It is not possible to invent missing data. No need to re-render the file, finding a good (before the truncation) copy of the file should be enough, I personally expect that the rendering tool correctly stored the file. If you lost this file, the only solution is to re-render the file, as the data is lost.

[X-Raym]

@JeromeMartinez

I thought there might be a tool which could define new end for the file (at the last readable data ) and update metadata accordingly.

Anyway, thanks for your infos and expertise.

For sure having good file right from the start would be the better way but sometimes there us just no way to get back.

Note that I use BWFMediaEdit CLI version mostly so other non CLI version might not work in my case. I have coded a lua script which is able to analyse wav file header and display metadata so I might go with this solution for the dump. But for metadata injection, it seems that only rerender can work.

[X-Raym]

@JeromeMartinez Seems that MediaTrace is able to dump xml file with the metadata even with broken file.

This is from a broken file:

<?xml version="1.0" encoding="UTF-8"?>
<MediaInfo
    xmlns="https://mediaarea.net/mediainfo"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="https://mediaarea.net/mediainfo https://mediaarea.net/mediainfo/mediainfo_2_0.xsd"
    version="2.0">
<creatingLibrary version="18.05" url="https://mediaarea.net/MediaInfo">MediaInfoLib</creatingLibrary>
<media ref="G:\REAPER Scripts\RPP-Tests\BWF\AutoRaceProStocks_6043_73 - Copie.wav">
<track type="Général">
<AudioCount>1</AudioCount>
<FileExtension>wav</FileExtension>
<Format>Wave</Format>
<FileSize>5978664</FileSize>
<Duration>20.426</Duration>
<OverallBitRate_Mode>CBR</OverallBitRate_Mode>
<OverallBitRate>2341590</OverallBitRate>
<StreamSize>95784</StreamSize>
<Title>Series 6000 Extension I</Title>
<Title_More>All Sound Effects &amp; Music are Copyright Sound Ideas - All Rights Reserved</Title_More>
<Album_Performer>Series 6000 Extension I</Album_Performer>
<Track>Series 6000 Extension I</Track>
<Track_More>All Sound Effects &amp; Music are Copyright Sound Ideas - All Rights Reserved</Track_More>
<Track_Position>0</Track_Position>
<Original_Album>www.sound-ideas.com</Original_Album>
<Performer>Sound Ideas</Performer>
<Director>Sound Ideas</Director>
<Producer>Sound Ideas</Producer>
<Genre>AUTO, RACE, PRO STOCKS / AUTO, RACE, PRO STOCKS</Genre>
<Description>AUTO, RACE, PRO STOCKS - SINGLE STOCK CAR:  PASS BY</Description>
<Original_Released_Date>2016</Original_Released_Date>
<Recorded_Date>2016 / 2016-05-20</Recorded_Date>
<Encoded_Date>2007-05-03 14:09:13</Encoded_Date>
<File_Created_Date>UTC 2018-06-03 17:25:25.256</File_Created_Date>
<File_Created_Date_Local>2018-06-03 19:25:25.256</File_Created_Date_Local>
<File_Modified_Date>UTC 2018-06-03 17:23:20.174</File_Modified_Date>
<File_Modified_Date_Local>2018-06-03 19:23:20.174</File_Modified_Date_Local>
<Encoded_Application>Soundminer</Encoded_Application>
<OriginalSourceForm_Name>Series 6000 Extension I</OriginalSourceForm_Name>
<Owner>Series 6000 Extension I</Owner>
<Copyright>2016 Sound Ideas (Series 6000 Extension I) </Copyright>
<Cover>Yes</Cover>
<Cover_Type>Cover (front)</Cover_Type>
<Cover_Mime>image/jpeg</Cover_Mime>
<Comment>AUTO, RACE, PRO STOCKS - SINGLE STOCK CAR:  PASS BY / AUTO, RACE, PRO STOCKS - SINGLE STOCK CAR:  PASS BY</Comment>
<extra>
<Producer_Reference>Series 6000 Extension I</Producer_Reference>
</extra>
</track>
<track type="Audio">
<Format>PCM</Format>
<Format_Settings_Endianness>Little</Format_Settings_Endianness>
<Format_Settings_Sign>Signed</Format_Settings_Sign>
<CodecID>1</CodecID>
<Duration>20.426</Duration>
<BitRate_Mode>CBR</BitRate_Mode>
<BitRate>2304000</BitRate>
<Channels>2</Channels>
<SamplingRate>48000</SamplingRate>
<SamplingCount>980448</SamplingCount>
<BitDepth>24</BitDepth>
<Delay>0.000</Delay>
<Delay_Source>Container (bext)</Delay_Source>
<StreamSize>5882880</StreamSize>
<StreamSize_Proportion>0.98398</StreamSize_Proportion>
</track>
</media>
</MediaInfo>

As you can see there is some metadata infos there. This is pretty nice, but doesn't solve the data injection problem (which was what I wanted to do in a first place). For MediaConch, the file appears to be valid. So it is just BWFMetaEdit who can't process it (extracting or injecting metadata).

[X-Raym]

Just an extra post for clarification. My case was the following:

1. I have a file, which appears as truncated in both GUI and CLI version of BWFMetaEdit
2. I wanted to edit the file Metadata (fields like description, Originator etc) with the BWFMetaEdit CLI version.

But it breaks.

I tried with other BWF injection solution like wavefile and this works. It sure didn't reinvent the missing data (if there is missing data, maybe it is just wrong metadata infos), but it made BWFMetaData readable and writable again in BWFMetaEdit.

So, a way to fix metadata sounds possible, having that with BWFMetaEdit CLI version would be nice !

[JeromeMartinez]

I thought there might be a tool which could define new end for the file (at the last readable data ) and update metadata accordingly.

Usually (important: not the case here, it is just an explanation about the idea behind the current way it is working), a truncated file is due to bad transfer, data is missing. It is doable to "define new end for the file", BUT it would just be hiding your issue, this is like a lie to whoever is interested in knowing if the file is wealthy. We don't want to create tools helping in hiding issues, as we are interested in fixing the issue (by getting a wealthy file), not hiding it. It is better IMO to store the file as is in that case, rather than hiding the issue by changing the header (the file is missing content, and you accept it, OK so no need to hide that)

Seems that MediaTrace is able to dump xml file with the metadata even with broken file.

You provided a MediaInfo XML, not a MediaTrace one.

Checking your file, issue is not the one @dericed thought about, the message was misleading (reason MediaConch ignored it, as it shows an issue only if the file is shorter than expected. Note that MediaConch does not have a WAV file conformance checker, the message "valid" should be "not applicable", something we need to fix in MediaConch).

MediaTrace output:

000000 WAVE (12 bytes)
000000  Header (12 bytes)
000000   Name:                                 RIFF
000004   Size:                                 5978656 (0x005B3A20)
000008   Real Name:                            WAVE
5B2CDE  _PMX (3402 bytes)
5B2CDE   Header (8 bytes)
5B2CDE    Name:                                _PMX
5B2CE2    Size:                                3658 (0x00000E4A)
5B2CE6   Unknown:                              (3394 bytes)
(...)

You can see that there is a problem: WAVE chunk says that end is at 0x5B3A28 (8 bytes of start of header + 0x005B3A20), _PMX chunck says that end is at 0x5B3A30 (offset 0x5B2CE6 + size of 0xE4A), 8 bytes later --> This is incoherent, the file is buggy.

In that case, currently BWF MetaEdit prefers to not edit the file, as there is a risk that something went wrong somewhere (this is not a file transfer issue, as the WAVE chunk header is coherent with file size), and it is more development work to edit other fields (e.g. bext chunk) while keeping such bug during rewrite (in order not to hide the issue). @dericed, a quick and dirty fix could be to ignore such bug in a file and "correct" it silently (1 line patch), but I am personally not in favor of such solution because I don't like to hide the issue in the file.

Checking the file, the XMP content is with spaces at the end, so low risk that some data was lost (just missing 8 spaces?), so we could add a check of such buggy file and fix it, this is just not yet implemented.

In other words, your file is a scenario of non conforming file we chose to avoid in our development, as it was not requested by the sponsor. Support of such buggy file (with different scenarios in order to be sure that BWF MetaEdit does what the user expects, whatever is the user expectation) could be added on request.

summary: I see 2 issues:

• support of buggy sub-chunk sizes in a chunk. I would assign this ticket to this issue. I change the title to "Support edition of some non conforming files" and add a "need sponsor" label. @dericed don't hesitate to change my change if you dislike it much.
• support of an optional fixer for buggy sub-chunk sizes in a chunk.

I tried with other BWF injection solution like wavefile and this works.

If it fits your needs, that's fine. BWF MetaEdit is just (currently) not adapted to your need of edition of such buggy file, we may have just have different priorities (my priority is to not miss a bug in a file, and to not hide it silently during rewrite, as wavefile does).

[X-Raym]

@JeromeMartinez Thanks for this detailed report and your time on this !

We now know what could be done to fix such issue 👍

Cheers !

[wpchase]

This is happening to me as well, running BWF MetaEdit 1.3.8 on Ubuntu 16.04.6 only with RF64 .wav files (> 2GB). The odd thing is that it only occurs within the context of a bash script. When I run the same BWF MetaEdit commands standalone, it works fine, and the file is modified without being truncated.

[g-maxime]

@wpchase I tried to reproduce the bug, without success.

You are using an i386 or amd64 version of bwfmetaedit ? This happen for all 2GB+ file or only some ?

You can try this snapshot (builded with -DFILE_OFFSET_BITS=64): amd64 i386

[wpchase]

@g-maxime thanks for your reply - I've been testing some more.

I think we are using the i386 version, need to doublecheck w/sysadmin.

All files are 2GB+.

When I run 1-4 files, they save fine. But a larger batch, of say 15, then they all become truncated. This is also true with the Windows GUI 1.3.2, pointing to files on the same server.

When I tried to modify 8 2GB+ files on my local drive with the Windows GUI, they all became truncated.

[wpchase]

[wpchase]

@g-maxime I tried the build you shared, and still got the invalid Wave: truncated error after BWF MetaEdit modified the file.

[g-maxime]

@wpchase Thanks for your informations, I can reproduce this bug. Now I am investigating on it.

[wpchase]

@g-maxime any updates on this?

We've been doing a workaround for now, re-saving our >2GB files. When they originally are recorded in Wavelab, Wavelab writes any file over 2GB as RF64. There is no other workaround than to batch re-encode as a regular Wave, and then run through the script that uses BWF MetaEdit.

[JeromeMartinez]

@wpchase I am the one who is not validating @g-maxime patch proposal, still on my free support todo-list. Note that free support has less priority than support packages or our psonsors, if you need faster fixes please consider to be a MediaArea sponsor.

[privatezero]

Hi - running into a related issue. Using version 1.3.8 in Ubuntu 18.04, an RF64 file will open and already embedded BEXT metadata will be viewable , but when any changes are attempted to be saved (such as updating a field or adding an md5) BWF Metaedit seems to create a truncated file. Any future attempts to open that file in the GUI will then fail.

Since this is essentially BWF Metaedit taking valid files and breaking them without warning, I thought it was different enough from the prior issues to rate reporting, but my apologies if this still falls into the same category of needing sponsorship!

[g-maxime]

@privatezero Hi,

The original bug is now fixed in the master, please try with a recent snapshot CLI, GUI

[privatezero]

Thanks @g-maxime - I tried latest snapshot and everything is copacetic now. Love the new logo also!

[dericed]

Hi @X-Raym, there's a new release now at https://mediaarea.net/BWFMetaEdit, which includes more verbose reporting on truncation, so in your example file now it states: AutoRaceProStocks_6043_73.wav: invalid Wave: truncated (The _PMX chunk should be 3658 bytes, but is 3394 bytes.)

This helps clear up that the audio (the 'data' chunk) is okay but that there was an issue with how the XMP chunk was written. Let us know what you think and feel welcome to close the issue if the behavior of the new release addresses the initial concerns of this ticket.

[X-Raym]

@dericed Many thx for the info and update ! Nice to have more detailed report. Considering this error, it seems it could be fixed just by updating the chunk size reference in chunk header to the expected value. But I guess this is out of scope of BWFMetaEdit to fix such wave file bugs, and bypassing it may lead to further problem, so I guess just having more detailed report will be fine. I close this for now and reopen of needed. thx !

[JeromeMartinez]

I guess this is out of scope of BWFMetaEdit to fix such wave file bugs, and bypassing it may lead to further problem

It could be in the scope, but we can not do all in 1 sponsorship, so out of the scope for the sponsorship for fixes and updates we currently have, but could be in the scope if there is another sponsorship ;-).