Scan considerations. Related #airsonic/airsonic#1708, #1736. Prerequisites: #1909, #1922, #1932, #1925

Problem description

At some point the behavior of the test changed even though I didn't change the logic. I have a guess, but I'm not sure.

[Guess] FirstChild frequently used during scanning depends on Java's NIO API or POSIX. So there may be cases where it's not perfect. If we want to ensure correctness, logic like using Firstchild should be moved after the logic that determines the current sort key. (Sorting logic when reading directories and files with NIO is unacceptable)

Steps to reproduce

Flaky like #1645, #1560 may occur in very rare cases (libraries with insufficient tag maintenance). Modern Windows and Linux behave very similarly, but Windows Server can behave quite differently under certain conditions.

This series of issues was not observed until 20220410.1 and has been observed since 20220425.1 on Windows Server.

(Since Windows Server is not used much...) The substantive impact is considered to be low, However, Scan flow would be more complete if the result was unified on all OSs.

Solution guidelines

The cause of the glitch has not yet been determined. The verifications scheduled for v111.6.0 are as follows:

[x] #1909. A conventional scanning process would be split into subclasses.
[x] #1925. Furthermore, one of the subclasses, the scan procedure class, will further subdivide the processing.
- This was originally intended as a necessary design change regardless of the existence of this issue. This is necessary for improving testability and for designing parallel tasks. As part of this, the timing of the process of creating sort keys will change.
[x] Validate again using the test case where the problem occurred on Windows Server.
- Test results have changed over time. Requiring a comparison before and after changes to the scanning process

Research

https://github.com/tesshucom/jpsonic/issues/1840#issuecomment-1399908058

Test cases with different test behavior depending on the OS can be classified into two types based on their causes.

A. There is a problem with the implementation, and the operation specification originally depends on the OS

It was a known issue. The parsing order depends on NIO's file list order. So it's OS dependent.
In #1925, differences in NIO file list behavior across platforms will be removed.
- This should eliminate most glitches in your test cases

B. Glitch that is completely unexplained for now

1645

MediaScannerServiceImplTest$GetChildrenOfTest#testBehavioralSpecForTagReflesh()

Perform tag edits on files after scanning and check whether the results are properly reflected after rescanning. At this time, instead of actually opening the file and rewriting the tag, it is reproduced by overwriting with the edited file. (Because that operation is a common use case.)

Guess

The test content of rewriting a file's tag and rescanning it (that is, twice) must be assumed to be relatively fat test! (And since Lucene writes are asynchronous, it's a pretty tricky test case.) And since it goes through Git's clone, it is not possible to manage timestamps for filesfor tests on Git. File timestamps also depend on file copying done by Java's NIO dependencies during test execution. The applicable test case issue was initially only Windows Server 2022 & JDK11, but now it is also seen in Ubuntu & JDK11. It has been tried quite a few times over a long period of time, but has never been observed in JVMs since JDK11. Initially, it was speculated to be a topic of the OS timestamp specification. (Windows Server timestamp specifications differ depending on the version. (Even client Windows OS, not server, has different specifications depending on the version). But now there is a suspicion of incompatibility on a per-JDK basis. As of now, Windows testing will not be done from JDK17 onwards. Because Docker is provided with JDK17. Windows & JDK17 combination will be added when LTS of JDK11 expires

Fatal or not

**As far as we know that**, this may not be considered a problem. - I've never seen this behavior on my local JDK11. Also, I haven't seen any bug reports for this on Airsonic and Jpsonic. Only Maven test case execution on Github Action reproduces the problem reliably. It's probably not serious. It's not a product problem and may be considered a problem specific to testing in a particular environment. - Jpsonic provided user-specified options instead of timestamp-dependent update detection. In other words, A workaround is already provided if something goes wrong. - If you use special storage like Google Drive, the File Date Dependent Scanning may not be available. In that case, on the Jpsonic side, it is necessary to specify the album for tag update before scanning. - It's annoying, but for music libraries, tag updates aren't an action that happens very often. (Actually, most of them are only cumulative new additions. If not, the libraly management method itself should be reviewed.) The workarounds already provided should be sufficient for practical use.

It will be handled as follows.

A : In #1925, Root cause is eliminated. The test case is rewritten to the original expected behavior.
B : Suppress MediaScannerServiceImplTest$GetChildrenOfTest#testBehavioralSpecForTagRefresh() from being executed when verifying JDK11.

By the way, B's suppression may be removed when JDK11's LTS expires. No more tracking will be needed. This issue will be closed when pull request on #1925 is merged.

maven-enforcer-plugin bumped to 3.2.1 at b4e82696f465c43416cde6e51bca428c22e8a219. Because it was affecting some test cases badly. (Once every few years something like this happens to the maven-enforcer ... )

2041 made the following related fixes:

A. OS dependent implementation removed

In parsing albums, the legacy server used "First child" files that relied on NIO as a source of information. Jpsonic will assign a sort key to the children before parsing the album and will use the first fetched child. This is done in both File Structure/ID3.
Basically, the track number is prioritized, and the path comparison is performed if it is not available. However, since Collator of Java is used for path comparison, there is no difference in results depending on the OS.

B. Album rescan specification changed

Still unsure about the inherent file glitches that might exist in Github Actions.
However, since #2041, directory timestamps are no longer used to determine album reparsing.
- When a song file update is detected and reflected in the database, FAR_FUTURE is registered in the ChildrenLastUpdatedDate of that album. (indicates that the child has changed and needs to be parsed again)
- In a later process, all albums with ChildrenLastUpdatedDate == FAR_FUTURE are reparsed.

What does that mean...

As Subsonic users know well, changing files often didn't change album data. In such cases, a well-known workaround is to run the touch command on the filesystem to update the timestamp of the album directory. One of the so-called bad know-how.

#2041 is expected to improve such cases. This is because it no longer depends on the OS's directory timestamp specification.

tesshucom / jpsonic

Rescanning tags may fail on Windows Server 2022 #1840