Open mikaelmechoulam opened 1 year ago
Hi @mikaelmechoulam can you confirm which mechanism you're using when you see the slow performance? Is this the new Java API or one of the other command-line mechanisms? Could you give a command or code snippet to represent what you're using so we could look to reproduce it.
@steve-daly We reproduced both by calling directly matchBinarySignatures and with the new Java API, with the following code: final DroidAPI droidAPI = DroidAPI.getInstance( Paths.get("../droid/DROID_SignatureFile_V90.xml"), Paths.get("../container-signature-20220704.xml")); final List<ApiResult> results = droidAPI.submit(Paths.get("../hugezip.zip"));
Thanks @mikaelmechoulam That Java API has only been added for internal testing at this stage so we wouldn't recommend using it in any production applications yet. We'll look into this performance issue though.
Note that our main version is still 6.5.2 currently and although 6.6.0 is visible on GitHub we'll hopefully be releasing 6.6.1 shortly to fix some issues noticed, and that version will be accompanied by some additional information about all the new features since 6.5.2.
@steve-daly Thanks, but please note we reproduce the issue in earlier versions (6.5.2 for example) and without using the API (by calling matchBinarySignatures )
Thanks, we'll take a look. Just to mention that you're using an unmatched pair of signature definition files there. V90 is from 2017 and should match container 20170330, whereas the container sig used (20220704) should match binary v107. That's not going to explain your issue here, but it may cause other undefined identification behaviour.
@steve-daly We have more details( we're still investigating to help you). Indeed, the main factor seems to be the combination container/binary file. It has nothing to do with a difference between DroidUI and a call to Droid libraries. But surprising as it may seem, for example if we use a "latest" combination container-signature-20221102 and binary v110, it has this huge bandwidth (read throughput = around the size of the file !). It is even worse with the combination container20220704-binary v106 (the file is read around 4 times !). We could have expected the contrary. Strangely some other combinations show a very low bandwidth (only a few kb) with a success. But in fact how do we know what is a "proper" combination between container and binary (only the time of the year?). Where is the link between the two files ? And why some combinations have this effect on bandwidth ?
Thanks @mikaelmechoulam it's interesting to see different behaviour with different signatures. If you look in the header of the binary signature XML file you should see a creation date which should point to the appropriate container signature that accompanies it. That said, I think I can see a discrepancy with the current signature set, which I'll look into now.
ZIP files specifically will trigger the container signature logic, where it tries to look for other files which use the ZIP format under the hood (such as Word Documents) and this means many more comparisons, but maybe there's something unexpectedly varying and causing this behaviour, so we'll look into this.
@steve-daly Here is an example of comparison of read bandwidths for a 4 Gb file.
We are annoyed because today we have to recommend our customer to use a newer container file with his older signature file, OR to modify this signature file manually (which is not very good in term of future upgrade). I have changed the title of the ticket to be more accurate.
@mikaelmechoulam @steve-daly really interesting issue. I was wondering if the same behavior is seen in Nanite which is a third party tool by Andy Jackson at the British Library that uses DROID and possibly implements something close to what you're doing Mike? https://github.com/openpreserve/nanite (Not sure how easy it is for you to measure the same effect?). If the problem doesn't occur then maybe Nanite also demonstrates an implementation pattern you can adopt to avoid this?
RE: The combinations table above - clearly there are more file formats in both sets of signature file, i..e the latest container signature and non-container signature. Perhaps the number of permutations of potential match in the non-container signature are increasing with v111 and 2023-03-07 and amplifying the behavior?
RE: Behavior in the UI, you may need to check too the maximum bytes to scan is set to -1 to recreate the issue in the first place reliably - Image below shows the default 65535:
Hi @mikaelmechoulam I suspect that the forth item in your comparison grid where you have unmatched Binary and Container signature files is probably just causing DROID to not run detection fully, rather than somehow working perfectly but faster.
It's interesting to see the correct pairing for v90 is also showing the behaviour (although not as bad as with v106) so it's not a recent introduction in the signatures, which I was going to check for.
Could you try a binary-only match, for example by using 6.5.2 and running 'no-profile mode' on the command line by using the -Nr switch (followed by the file/folder to check) and passing the binary signature file location using the -Ns switch. That would avoid all the checking that happens following a match with ZIP format
I might be misunderstanding how DROID works, but I think scanning the whole thing is the expected behaviour for ZIP files, and has been for some time. I wrote up my reasoning in this blog post rather than take up space here. But briefly: there are binary signatures that use an unbounded wildcard after the ZIP signature (e.g. 504B0304*4D45...
), and as no byte limit is set, I'd expect a full file scan to happen for all ZIP files, multiple times (once for each binary signature of this form).
If that's right, then I think @mikaelmechoulam may be seeing a cache exhaustion effect, where for some reason the system DROID is running on no longer has enough free memory to effectively cache large files.
EDIT 2023-03-22: I don't want to take any more space up in this thread, but just wanted to thank everyone for a very helpful discussion which has improved my understanding of how DROID works. There's some details here if anyone's really that interested!
@mikaelmechoulam having read the blog that @anjackson wrote we made a draft modification to the binary signature, that if it works well we can hopefully add to the next official PRONOM release. It seems to speed up DROID internally for us.
Does using the signature in the zip file, alongside the latest container signature speed up DROID processing? Just to confirm the binary signature in this file is just for testing purposes and not an official release.
The difference in this signature is that we have completely removed the binary signature for fmt/161 and x-fmt/412 as they already have container signatures. If it works well then we will have to do some tests to see if the identification is affected by not having the binary signature, which it doesn't seem to be so far.
The plan is to do a bit more research into how many file formats are identified by both container and binary signature and whether this is required in each case. Especially those with the * wild card, which @anjackson is completely correct, definitely slow down processing.
Here is a rough mapping of each binary signature and the corresponding container signature:
<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40">
Binary Signature | Container Signature -- | -- V46-59 | 04 February 2011 V60-61 | 11 June 2012 V62-65 | 28 August 2012 V66 | 18 December 2012 V67 | 26 February 2013 V68-71 | 01 May 2013 V72 | 12 November 2013 V73-74 | 27 February 2014 V75-77 | 17 July 2014 V78-80 | 23 September 2014 V81 | 18 February 2015 V82 | 27 March 2015 V83 | 17 December 2015 V84 | 21 January 2016 V85 | 29 June 2016 V86-87 | 27 July 2016 V88 | 27 September 2016 V89 | 08 March 2017 V90 | 30 March 2017 V92 | 20 September 2017 V93 | 30 November 2017 V94-95 | 20 September 2018 V96 | 21 January 2020 V97 | 01 October 2020 V98-99 | 27 October 2021 V100-102 | 16 December 2021 V103-105 | 11 March 2022 V106-107 | 04 July 2022 V108 | 05 September 2022 V109 | 02 November 2022 V111 | 16 March 2023
Hello, When identifying a ZIP file which matchBinarySignatures (Droid 6.6), a profiling tool shows that the ZIP file seems to be read fully one or several times (this ratio depending on the version of the chosen signature file). When attached to the DroidUI tool (and when we choose not to analyse content of archive files, of course), the profiling tool shows a very low bandwidth. The max bytes scan is set to the same value (maximum value). For example for a 4Gb file, when using the code the 4Gb are read, when using DroidUI only a few kilobytes. We tried the same comparison with a MP4 file, there is not this discrepancy. Do you have an explanation ? Thanks,