1T is struggling with External HDDs or/and High Capacity HDDs?

[rosgr100]

HI @Marekkon5.

It seems that 1T is having difficult times with external HDDs and/or high capacity HDDs, auto-tagging files from internal (less rpm, slower) HDD takes almost half or even 1/3 of the time needed in comparison with the external HDDs (or maybe high capacity HDDs?). Also sometimes, only when i use 1T's autotagging i get several HDD disconnections.

1t is unable to read or write faster than 1-3MB/s in an External HDD

I've conducted several tests, different drives, cables, ports, PCs etc, also i've checked the HDDs for bad sectors, health issues, or errors and eveything was OK.

WD ELEMENTS 12tb

[rosgr100]

Forgot to mention that i used the same autotagging settings on the testings above.

[Marekkon5]

Thanks for the tests, I gotta try myself and check if I didn't forgot to use a buffered IO somewhere. However maybe it's just because 1T does a lot at once - can you try with lesser threads? It should access the HDD less. Thanks

[rosgr100]

I've already tried 2-4-8-16-24 threads, same results!

Thanks for the response!

[Marekkon5]

I've checked the code - nothing out of the ordinary, tried comparing my external SSD and got ~25% slower times. However I do believe it's a fault of USB - from experience I know that USB HDDs suck with random reads/writes (for example booting from external HDD takes longer than from the same HDD over SATA). And your tests confirm that too - the random reads/writes speeds are very low, and 1T writes only few KBs per file (the tag). Since I don't have external HDD at this moment, @bascurtiz will do the tests. Thanks

[rosgr100]

I've done a test with Beatporttagger (which was always a lot faster than 1T) and it doesn't suffer from the same problem.

[Marekkon5]

There are few small differences between Beatport Tagger and 1T, which make it use a bit more network:

• 1T fetches 150 results per page (Beatport Tagger only 25)
• Some tags (catalog number, album artists, track number, isrc, styles) aren't on the search results page, so they require 1 or more additional requests (= slower, more network usage)

However that doesn't really explain the disk speeds.

[Marekkon5]

In the latest commit, I've changed the results per page to be 25 (same as Beatport Tagger). It is significantly faster now (especially if you disable the tags I mentioned above too), so let me know if it's better for you.

In theory it will slightly decrease the match rate, however in my small (200 tracks) test it didn't change, and we will do more tests in the next days. However you can just change the Max Pages option to 6 to get the exact same results as before.

You can get the binary in the Actions tab. Thanks

[rosgr100]

Ok, thanks, i'll give it a try to see if this will resolve the issue.

By the way, i've made some extra testing, for the internal drive this time

1T was able to peak read/write speed at ~15MB/s BTTagger is able to read/write at even 50MB/s (could not capture it, i wasn't fast enough)

[rosgr100]

Hi @Marekkon5 i did some extra testings with the latest commit, didn't change much regarding the read/write issue.

What i found out though, is that if you select traxsource as the tagging platform, the problem goes away. Read/write speed is peaking somewhere between 80-100MB/s.

2 16 threads tests

24 threads test

So, i assume that the problem is somewhere in beatport's script.

[Marekkon5]

I think it's Beatport being slow, rather than the disk itself. Which tags do you have enabled? Can you send the log file too? Thanks

[rosgr100]

All, except Artist, Title, Duration and ISRC, usually all, except Artist and Title.

Beatsource -on a small test- seems problem free too, although it wasn't able to identify many tracks.

However, Beatport being slow wouldn't cause the same problem to BTagger too or with internal HDDs?

LOG https://mega.nz/file/ixgnCSbB#CHAv1G79s3NQA33KBKuXNqa7lQacUbKKHCz5McbT7dk

Link will expire soon, if you didn't download it, leave a note.

Forgot to mention that i also tried to only select the same tags as in BTagger, with the same -slow- read/write results.

[Marekkon5]

Well, the issue isn't slow reads/writes to your HDD, but Beatport / Beatsource itself. We've tested it in past, and even now - if you run the exact same test in different times of the day, you will get different results - it's the Beatport servers itself being slow and there isn't much we can do about it (maybe allow even more threads).

As I mentioned earlier - using tags which aren't in BPT causes 1T to be slower - catalog number, album artist, styles, track number, isrc, or the use track id/release id to match feature. It is because they require 1T to also fetch subpage(s) from Beatport - more strain on their servers, more network usage - slower. Without those 1T should match, or even be faster than BPT.

I was asking for log to see if it does those extra requests for you or if you really have the latest commit, since it is significantly faster for me. However sorry, it was late last night, so I couldn't download it, the link is now expired. Could you send again please?

As for Beatsource - again, their library is rather small, so not really an 1T issue. However if it can't match a track that IS available on Beatsource, feel free to send here then title - artist and I'll try to fix.

Thanks

[rosgr100]

Hi @Marekkon5, I understand what you are describing about Beatport servers being slow sometimes but this can't explain why every autotagging process -using Beatport as source- in an internal HDD takes almost 10%-20% of the time that it takes in an external HDD and this is happening everytime i try, no matter what tags or settings i'm choosing.

Take a look (16 threads, 1 page, all tags selected except artist and title, same settings on both runs, same folder/files and almost at the same time)

Internal HDD: tagging took 6:44 minutes, read/write speeds are between 20-50MB/s, tagging starts almost immediately.

External HDD: tagging took 43:58 minutes , read speed (which seems to be the problem and not write speed) is stuck somewhere between 1-2MB/s usually at 1.4MB/s, first tagged track after 2:46 minutes, and at 6:50 minutes 1T was able to tag only 30 tracks, while in the internal HDD, at the same time, 1T was able to tag all the tracks.

PS. I mentioned Beatsource just to indicate that it seems to be problem free regarding the slow read speeds, not because i was trying to state that 1T is not able to match tracks which are available in it's library, i haven't test that thoroughly.

Log (last two runs) onetagger.log

Thanks!

[Marekkon5]

Hello, to be perfectly honest with you - we're out of ideas and can't really reproduce it, and I don't see anything wrong with your setup either. While it makes sense for external HDD to be way slower with random reads/writes, I don't think that much of a difference is normal. Maybe you can check resmon while tagging to see if there are any weird things going on?

Tomorrow I'll make a branch with a bunch of benchmarking and debugging info enabled for you to check to see if we can pinpoint what's the slow part.

Sorry for the inconvenience and thanks for more tests.

[rosgr100]

Probably the problem is on my side...

rasmon did the trick.

From what i can say it was either WDDriveService, EBCClient (EaseUS Backup Center Client) or GsServer (GoodSync Server). I suspect that the guilty one is the WDDriveservice (probably all WD external HDDs will suffer), because i tried a Seagate one (5400rpm) and it was problem free.

i have to further test to identify which one of the three services is responsible for the problem, but as i said guess that the one is the WDDriveService (people, turn this off).

I'll be back tommorow to report what the outcome will be.

SORRY and THANKS for the help.

Too soon to judge. Same problem again. Can't figure out what's happening.

[rosgr100]

Hi @Marekkon5 i think i sort that out. The root of the problem is the album art tag, deselecting it resolves the issue, which is not an ideal solution. Now, selecting album art and disabling the Overwrite option is making the read\write issue to go away. So i suspect that the solution to the problem lies somewhere there, in the overwriting procedure.

I observed a weird behavior, when album art tag is selected and overwrite is enabled, 1T is erasing all the tags, even those which are not selected and it writes them back afterwards.

Thank you!

[bascurtiz]

It's strange, cause we both can't replicate the slow reading/writing or tagging. In this video demo, I've made sure those are all ID3v2.3 before tagging. Then making them ID3v2.4 if they get tagged. Also removed the Album art (not that both matter for the processing speed on my end). Writing to external 12TB HDD WD, but don't have their bloatware installed.

Maybe you spot a different setting than what you are using?

What I do notice is that your so called Active Time is on 100%, while mine only reaches that 2-3 times for a short period. I only have a few bumps of spikes, but your chart looks like filled to the max for longer periods of time.

See example: https://youtu.be/nPcoKcY71jA

[rosgr100]

Hi @bascurtiz, thank you for the video. Following your steps and removing only the album art before the autotagging process, resolves the issue and i have almost the same active time stats as yours, i assume that due to my slower internet, it takes more time to download the tags so this is why i also have zero active times and higher spikes also.

Trying exactly the same without predefined album art being previously removed (same files)

[bascurtiz]

Ahh, I assume you had huge pictures in your album art, which caused the slow process then, while reading? Glad you found a work around for speeding things up.

[rosgr100]

The biggest one is 2MB just two covers, the rest we can call them normal size.

At least, yes, for now there is an easy solution.

[bascurtiz]

Hmmm, I'll check when I embed 1400x1400 pics prior and test again.

[rosgr100]

I removed only those too, same problem.

Could some .png covers be responsible for the problem? I'll try to remove those also to see if png files are creating the problem.

Update: NO, it's not .png covers.

[bascurtiz]

I've ran it on 1000 unprocessed tracks, where I kept the album art (1400px), but can't seem to replicate extreme slow tagging speed either. See demo: https://youtu.be/WCPDIBrr5WQ

[rosgr100]

Then, i have no idea why this is happening to me. I'll try to completely remove 1T and reinstall it.

[Marekkon5]

@rosgr100 Looking at your screenshot where you had 100% disk usage and no tracks found for almost 2 minutes - it looks VERY weird. Could you maybe check resmon again to see if there really isn't any service interfering? Or perhaps if you could send few tracks from the folder with original tags to see, if it is really issue of the tags themselves? Thanks.

[Marekkon5]

@rosgr100 As I promised earlier - I've made a test version which has a lot of benchmarking info spammed in the log. Could you please try it and send log with the issue present? You can get the binary from the Actions tab, it's called (DON'T USE) - Benchmark for #77. Thanks

[rosgr100]

HI @Marekkon5 thank you for all the effort you are putting on this guys. I don't know if this is normal, but as i already said above i think that for some reason the tracks are being "reconstructed" by the system, this is what i observed in resmon, i can't identify anything else, point me if there is something specific that i have to watch.

I'll try to upload some tracks later, unfortunately my upload speed is only 1mbit so it will take a while.

Thanks again, i'll run some autotagging processes with the test build (did i already say thank you?) and i'll send you the logs!

[rosgr100]

@Marekkon5 here is the latest log. onetagger.log

If you can identify where the problem is, pinpoint to me some of the tracks which are causing the problem and i'll upload the original files later today or tomorrow.

Thank you!

[Marekkon5]

Thanks, I'll take a look tomorrow at it.

[Marekkon5]

I've checked the logs and averaged the benchmarks, here are the raw results in milliseconds:

AudioFileInfo::load_file:  205
BP fetch_release:  2475
BP fetch_track:  4134
BP fetch_track_embed:  839
BP match by id:  0
BP match track:  12
BP search:  4319
Load file:  205
Match track (full):  11301
write_to_file: download and set art:  1107
write_to_file: get file  0
write_to_file: set normal:  0
Write track (full):  81889

As you can see, the obvious biggest time (82 seconds) is Write track (full) which should be just downloading album art & adding all the tags to a file. However write_to_file: download and set art takes only 1s, which means network isn't really a problem.

However I have no idea why the writing takes so long. Could you maybe try different format (FLAC) to see, if it's really a ID3 library issue? But since you mentioned before that Traxsource was fast, this seems REALLY weird to me, because then it would affect all of the platforms.

Thanks

[rosgr100]

Hi @Marekkon5, FLAC files are not causing any issue, i ran a small test and the process is super fast.

Regarding Traxsource now, everytime i get the same results, it seems that when i choose it as the tagging platform the problem disappears or it's not so obvious. Tagging Flac files, is 10 times faster than tagging mp3s. So probably you may are right about the ID3 library issue.

FLAC

Mp3

[bascurtiz]

@rosgr100 Can you bulk upload like 100 mp3 tracks to some free cloud storage like google drive, mega, zippyshare, etc.? ...since it now starts to look like there's something going on with your mp3's in particular (while with mine/ours we can't reproduce).

[rosgr100]

Hi @bascurtiz, sorry for my late response. I've totally missed your latest response. Unfortunately due to my slow upload speed it would take ages to upload a hundred of mp3's.

But, i believe that i finally found out what is the root of the problem. And of course, i was wrong that this is only happening when i use Beatport as source to autotag tracks

I believe that the problem is the way 1T is reading the ID3v2.4 tags. I've made a playlist which was only including tracks with ID3v2.4 tags, the problem appeared instantly. Converting the tags to ID3v2.3 with mp3tag, solved the slow read problem.

ID3v2.4

ID3v2.3

NO, IT'S STILL HERE...

[bascurtiz]

I tried with 100 tracks in MP3/320/ID3v2.4 with Beatport, all tags enabled/overwrite tags:

and tried with same 100 tracks but in MP3/320/ID3v2.3 with Beatport, all tags enabled/overwrite tags/ID3v2.4 disabled:

Due cache 2nd attempt was faster. But no oridinary differences in speed.

[rosgr100]

Yes, i thought that i sorted it out but as i mentioned above, i didn't.

Now, i observed something weird at the resmon. When i try to autotag files in my External HDD, there are multiple "instances" of 1T which are accessing the files, while at the same time the same happens with System process.

In the other hand, when i try to autotag -the same- files which are located in the internal HDD, there is no onetagger.exe process and the only process which is accessing the files is the system.

[Marekkon5]

That's normal however - 1T does use multiple threads to search Beatport and write to the files as well. You can try with 1 thread if the problem goes away. As for System vs 1T writing it might be how Windows works, but I have to google it.

[rosgr100]

Hi, i believe that i, somehow, sort this out.

I've done some more tests and while i was doing them i was also watching resmon to observe the differences between internal and external drive. Something that i observed, was the onetagger.exe was using the $Extend\$UsnJml on my internal drive but it wasn't on my external. i found out that my external was missing that file, so i've had to create it (https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/fsutil-usn). After the creation of the file, 1T was reacting in the same way as it was with the internal HDD, system.exe is the process that it's mostly gets the job done, onetagger.exe stopped accessing the files as it was doing it on the screenshots above.

I don't know how or if this is connected with 1T though.

Secondly, i found out that for some reason, selecting 800px as cover size -setting which i was using before- is creating the problem, but really rare now. 500px or 1000px does not.

Thirdly, reserving more padding space to the ID3 tag with mp3diags, is making the autotagging process faster and this also contributes to make the problem go away completely, although my files already had some free padding space.

By the way, i observed that after the autotagging process, 1T leaves zero padding to the ID3 tag frame, this creates a problem if you want to edit the tags furthermore, the process is slower because the file needs to be recreated, i think that it would be better to set it leave something like 2048 byte free padding space, as mp3tag does.

[Marekkon5]

Hello, in the latest commit I've added 2048 bytes of padding to the MP3 files. You can get the binary from actions tab to test it out.

As for the rest of the issue - well since we weren't able to reproduce it and you found a somewhat solution, I think I can close it now. Sorry for not being able to figure it ourselves. As for the 800px - I am not sure if I'll be able to fix it either, since we can't really reproduce it. Maybe it's just a fluke that it breaks on 800px and not on 1000px.

Thanks

[rosgr100]

Thanks for the addition @Marekkon5.

There is no reason to be sorry, you are doing an amazing job and, of course, you have helped a lot with my issue.

Once again, thank you for everything.

[rosgr100]

Hi @Marekkon5 I'm still facing the same issue.

I tried to record a BTagger and a 1T autotagging process with the Procces monitor. I have set the same tags and -i believe- settings in both apps.

One thing i'm seeing in 1T is that there are a lot of FAST I/O DISALLOWED messages.

Also BTagger sends many BUFFER OVERFLOW messages, something that is not happening with 1T.

You can find the reports (csv) of the 1T HERE and BTagger's HERE.

I had to split 1T's log in two parts because the file has too many rows. As you can see there is a huge difference in size -and rows- between the two logs.

I hope that this will help you identify what's the root of the problem.

Thanks again!

[Marekkon5]

Hello, from the log it seems that 1T is doing way too many small reads and writes (= too small buffering). In the latest commit I've added a big (1MB) buffer when reading ID3 (mp3/aiff) files, so that part should be fixed. However the writing part is more complex, because there is a bunch of safe mechanisms to not overwrite the audio data, so I'll have to consult the library's author to fix that.

As for the FAST I/O DISALLOWED - it just means that the data isn't cached, can be ignored. Not having BUFFER OVERFLOW is probably good.

Let me know if the latest build improves anything. Thanks

[rosgr100]

Hey, thank you for the fast response.

I tried the latest build and it doesn't seem to improve things. Probably, because of all these small writes, there is a huge strain on the drive and that's causing the issue.

Thanks again.

[Marekkon5]

The author has increased the buffer size to 64K, so it should be fixed in the latest commit. Let me know. Thanks

[rosgr100]

Thanks for all the time you have spend to resolve this.

You are amazing!