search

tesshucom / jpsonic

This is a repository for development. See https://github.com/jpsonic/jpsonic
GNU General Public License v3.0
13 stars 13 forks source link

Null pointer exception while scanning #2635

Open basos9 opened 2 months ago

basos9 commented 2 months ago

Problem description

While scanning my media folder for the first time, the scan seems stalled in gui. In the logs is see the errors noted below. And the java process seems quiet cpu and disk wise. So i assume it stalled. I left it there for some days and still the process has not finished.

Steps to reproduce

I dont know what exactly. I see theese in the logs.

2024-05-01 15:05:26.128  INFO --- com.tesshu.jpsonic.Application           : Starting Application v114.1.0 using Java 17.0.10 with PID 433405 (/media/aux/lib-jpsonic/jpsonic.war started by jpsonic in /)
2024-05-01 15:06:16.093  INFO --- c.t.j.service.VersionService             : Starting Jpsonic 114.1.0 (e95eee1f5597d17b1fac0bbd9fc3e794ebe48893), Java: 17.0.10, OS: Linux
INFO --- com.tesshu.jpsonic.Application           : Started Application in 66.591 seconds (process running for 69.5)    
2024-05-01 15:07:51.169  INFO --- c.t.j.s.s.ScannerProcedureService        : Scanned media library with 250 entries.
2024-05-01 15:08:11.455  INFO --- c.t.j.s.s.ScannerProcedureService        : Scanned media library with 13500 entries.
2024-05-01 15:08:12.453 ERROR --- jps-scan-task-pool-1                     : An error occurred in the pooling thread.
 java.lang.NullPointerException: Cannot invoke "java.lang.Integer.intValue()" because "result" is null
         at com.tesshu.jpsonic.service.metadata.ParserUtils.parseInt(ParserUtils.java:150) ~[classes!/:114.1.0]
         at com.tesshu.jpsonic.service.metadata.ParserUtils.parseYear(ParserUtils.java:165) ~[classes!/:114.1.0]
         at com.tesshu.jpsonic.service.metadata.MusicParser.lambda$getRawMetaData$9(MusicParser.java:161) ~[classes!/:114.1.0]
         at java.base/java.util.Optional.ifPresent(Optional.java:178) ~[na:na]
         at com.tesshu.jpsonic.service.metadata.MusicParser.getRawMetaData(MusicParser.java:161) ~[classes!/:114.1.0]
         at com.tesshu.jpsonic.service.metadata.MetaDataParser.getMetaData(MetaDataParser.java:59) ~[classes!/:114.1.0]
         at com.tesshu.jpsonic.service.scanner.WritableMediaFileService.applyFile(WritableMediaFileService.java:340) ~[classes!/:114.1.0]
         at com.tesshu.jpsonic.service.scanner.WritableMediaFileService.parseMediaFile(WritableMediaFileService.java:303) ~[classes!/:114.1.0]
         at com.tesshu.jpsonic.service.scanner.WritableMediaFileService.createMediaFile(WritableMediaFileService.java:290) ~[classes!/:114.1.0]
         at com.tesshu.jpsonic.service.scanner.WritableMediaFileService.createOrUpdateChild(WritableMediaFileService.java:229) ~[classes!/:114.1.0]
         at com.tesshu.jpsonic.service.scanner.WritableMediaFileService.updateChildren(WritableMediaFileService.java:176) ~[classes!/:114.1.0]
         at com.tesshu.jpsonic.service.scanner.WritableMediaFileService.getChildrenOf(WritableMediaFileService.java:243) ~[classes!/:114.1.0]
         at com.tesshu.jpsonic.service.scanner.ScannerProcedureService.scanFile(ScannerProcedureService.java:306) ~[classes!/:114.1.0]
         at com.tesshu.jpsonic.service.scanner.ScannerProcedureService.scanFile(ScannerProcedureService.java:310) ~[classes!/:114.1.0]
         at com.tesshu.jpsonic.service.scanner.ScannerProcedureService.scanFile(ScannerProcedureService.java:310) ~[classes!/:114.1.0]
         at com.tesshu.jpsonic.service.scanner.ScannerProcedureService.scanFile(ScannerProcedureService.java:310) ~[classes!/:114.1.0]
         at com.tesshu.jpsonic.service.scanner.ScannerProcedureService.lambda$parseFileStructure$3(ScannerProcedureService.java:276) ~[classes!/:114.1.0]
         at java.base/java.util.Optional.ifPresent(Optional.java:178) ~[na:na]
         at com.tesshu.jpsonic.service.scanner.ScannerProcedureService.lambda$parseFileStructure$4(ScannerProcedureService.java:276) ~[classes!/:114.1.0]
         at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) ~[na:na]
         at com.tesshu.jpsonic.service.scanner.ScannerProcedureService.parseFileStructure(ScannerProcedureService.java:275) ~[classes!/:114.1.0]
         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
         at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:351) ~[spring-aop-6.1.5.jar!/:6.1.5]
         at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:713) ~[spring-aop-6.1.5.jar!/:6.1.5]
         at com.tesshu.jpsonic.service.scanner.ScannerProcedureService$$SpringCGLIB$$0.parseFileStructure(<generated>) ~[classes!/:114.1.0]
         at com.tesshu.jpsonic.service.scanner.MediaScannerServiceImpl.doScanLibrary(MediaScannerServiceImpl.java:162) ~[classes!/:114.1.0]
         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
         at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]
2024-05-01 15:11:28.397  INFO --- c.t.j.s.PodcastScheduleConfiguration     : Is scanning. Automatic podcast updates will not be performed.

Maybe a malformed tag for year? JPsonic should survive from metadata irregularities.

System information

tesshucom commented 2 months ago

Thank you for your report. If a process that is not currently expected occurs, Jpsonic's scan will stop at that point. This is by design. 　(In exchange, you won't run out of memory or run out of CPU. We'll be able to recover by scanning again after fixing) Let's investigate the contents of your report. If reproducible data can be provided somewhere, we can fix it immediately.(Or, for future reference, could you please tell me the name of the client app used to edit the tag?) What do you think?

tesshucom commented 2 months ago

Maybe a malformed tag for year?

Yes. The ID3 specification defines it as follows:

TYER : The ‘Year’ frame is a numeric string with a year of the recording. This frames is always four characters long (until the year 10000).

In other words, the root cause is a bug with the tag editor you are using. Non-well-formed data is generated in your library from them. I am aware that this is not a bug in Jpsonic.

JPsonic should survive from metadata irregularities.

No. This depends on Jpsonic's specifications. We do not recommend using tag editors that contain bugs as a premise.

There are several types of approaches.

Perhaps users want some flexibility and friendliness. (Be somewhat flexible) I also believe that the specifications can be adjusted in this regard. In other words, you can change it to ignore the input and output a WARNING log.

However, please note that there are other ways of thinking, such as the following:

Therefore, this matter will be put on hold for about a week. If we don't get good cooperation, the issue will be closed. In that case, it may be fixed in the future, but it is considered not urgent. Its priority is lower.

If minimum reproducible data is obtained, it will be used in Projectct as testing data. These simulated data have been inherited from Subsonic and Airsonic, and have been further added to by Jpsonic. This is to avoid degradation failures that occur frequently on other servers. In this regard, amendments without test data will not be accepted.

If I were to do it alone, I would be manually creating data that doesn't exist 😄

basos9 commented 2 months ago

Maybe a malformed tag for year?

Yes. The ID3 specification defines it as follows:

TYER : The ‘Year’ frame is a numeric string with a year of the recording. This frames is always four characters long (until the year 10000).

In other words, the root cause is a bug with the tag editor you are using. Non-well-formed data is generated in your library from them. I am aware that this is not a bug in Jpsonic.

JPsonic should survive from metadata irregularities.

No. This depends on Jpsonic's specifications. We do not recommend using tag editors that contain bugs as a premise.

There are several types of approaches.

* Be strict : The policy is that higher-level specifications should be respected as much as possible. Input that does not follow this is an error.

* Be somewhat flexible : Ignore input that users often make mistakes and prompt them to correct with a warning. This policy is recommended for manual input that is generally performed repeatedly.

* Accept everything : Whatever it is, bring it into the System. Most of the time you'll end up with something terrible.

Yes I would argue for the "Be somewhat flexible" policy. And yet you should treat the input as unsanitized (I think everywhere in software). For example, my library has some history (from 2002), and contains stuff from friends, some from the internet, some other from ripped personal albums, etc. It is not possible to properly tag and sanitize every tag. And I have used some music library software. I had never this issue. Even not in airsonic. Yes you could count it in the contents of society;)

I would happily provide test data. If I could somehow make jpsonic to log which song was it about to scan just before the crash.

tesshucom commented 2 months ago

Thank you. I'm planning to provide a version that logs problematic files, but I won't be able to get started on it right away, so please wait a little longer 😣

I had some hope that I could make assumptions about the software used, but that doesn't seem to be the case. (Because if so, there was a chance I could use it to reproduce it without any hassle on your part.) If you combine multiple pieces of non-warranted software, the result will be non-warranted. Note that there is no validity to the claim that unwarranted software should always be guaranteed to perform out of specification. Jpsonic's policy is to deal with obvious irregular data on a case-by-case basis. We do not believe that compatibility with software that likes to work with dirty data is a prerequisite for good software.

If you're in a hurry... You can make the problem easier to understand by splitting the music folder. Jpsonic has been changed so that it is less burdensome even when using many music folders than a regular Subsonic compatible server. (Tests with large data use approximately 40 music folders) If you can separate the problematic music folders from the non-problematic music folders, you may be able to use only normal library data.

Yes I would argue for the "Be somewhat flexible" policy. And yet you should treat the input as unsanitized (I think everywhere in software).

Opinions may differ slightly on the sanitization. What you expect is availability for Irregular data relative to the specifications. (I'm not denying this. There's just a priority.) Perhaps this issue will provide a feature fix that's closer to what you expect. Or so I hope.

Conversely, the sanitization function for input values within the specification range is probably at the top level among OSS. I have never seen a media server that complements and naturally sorts characters from a variety of Japanese and English characters.

Originally ... Output bugs and bug data generated by output bugs. Availability of acceptance for it. The most reasonable idea is that bugs in the preceding process should be resolved first. From a fair perspective as an engineer. Essentially, the way to normalize problematic data is to re-edit it using a suitable editor. (Regardless of whether it is appropriate or not, this project is verifying the loading of SONY and Apple data.) On the other hands, I understand the situation your library is in. At least in this project, rescuing such music libraries has not been the primary goal until now. And not all software in the world will have such a polic. ... And standards for backward compatibility will change over time.

Tak-MK commented 1 month ago

I wanted to say that I've been having the same problem but sadly TRACE doesn't output which file has the issue, so I've isolated the culprit album by restarting the service each time it fails and moving a new album into scan.

Details of it: Folder name: PINOCCHIOP BEST ALBUM 2009-2020 「寿」 Track 1: 01 - 愛されなくても君がいる.flac Metadata of track 1 (only comments):

  comments: 7
    comment[0]: TITLE=愛されなくても君がいる
    comment[1]: ALBUM=PINOCCHIOP BEST ALBUM 2009-2020 「寿」
    comment[2]: DATE=null
    comment[3]: TRACKNUMBER=1
    comment[4]: ARTIST=ピノキオピー
    comment[5]: GENRE=j-pop
    comment[6]: CDDB=http://cddb.sevendats.com/j-music/%E3%83%94%E3%83%8E%E3%82%AD%E3%82%AA%E3%83%94%E3%83%BC%20%28PinocchioP%29/Compilations/PINOCCHIOP%20BEST%20ALBUM%202009-2020%20%E3%80%8C%E5%AF%BF%E3%80%8D/

Maybe the length of the CDDB tag is too long to store? Trace log:

jpsonic  | 2024-05-20 17:12:09.001 TRACE --- c.t.j.dao.base.TemplateWrapper           : Updated 1 rows
jpsonic  | 2024-05-20 17:12:09.001 TRACE --- c.t.j.dao.base.TemplateWrapper           : Executing query: [update media_file
jpsonic  | set cover_art_path = ?
jpsonic  | where path=?
jpsonic  | ]
jpsonic  | 2024-05-20 17:12:09.003 TRACE --- c.t.j.dao.base.TemplateWrapper           : Updated 1 rows
jpsonic  | 2024-05-20 17:12:09.004 TRACE --- c.t.j.dao.base.TemplateWrapper           : Executing query: [update media_file
jpsonic  | set cover_art_path = ?
jpsonic  | where path=?
jpsonic  | ]
jpsonic  | 2024-05-20 17:12:09.006 TRACE --- c.t.j.dao.base.TemplateWrapper           : Updated 1 rows
jpsonic  | 2024-05-20 17:12:09.010 ERROR --- jps-scan-task-pool-1                     : An error occurred in the pooling thread.
jpsonic  |
jpsonic  | java.lang.NullPointerException: Cannot invoke "java.lang.Integer.intValue()" because "result" is null
jpsonic  |      at com.tesshu.jpsonic.service.metadata.ParserUtils.parseInt(ParserUtils.java:152) ~[!/:114.1.1]
jpsonic  |      at com.tesshu.jpsonic.service.metadata.ParserUtils.parseYear(ParserUtils.java:167) ~[!/:114.1.1]
jpsonic  |      at com.tesshu.jpsonic.service.metadata.MusicParser.lambda$getRawMetaData$3(MusicParser.java:161) ~[!/:114.1.1]
jpsonic  |      at java.base/java.util.Optional.ifPresent(Unknown Source) ~[na:na]
jpsonic  |      at com.tesshu.jpsonic.service.metadata.MusicParser.getRawMetaData(MusicParser.java:161) ~[!/:114.1.1]
jpsonic  |      at com.tesshu.jpsonic.service.metadata.MetaDataParser.getMetaData(MetaDataParser.java:59) ~[!/:114.1.1]
jpsonic  |      at com.tesshu.jpsonic.service.scanner.WritableMediaFileService.applyFile(WritableMediaFileService.java:340) ~[!/:114.1.1]
jpsonic  |      at com.tesshu.jpsonic.service.scanner.WritableMediaFileService.parseMediaFile(WritableMediaFileService.java:303) ~[!/:114.1.1]
jpsonic  |      at com.tesshu.jpsonic.service.scanner.WritableMediaFileService.createMediaFile(WritableMediaFileService.java:290) ~[!/:114.1.1]
jpsonic  |      at com.tesshu.jpsonic.service.scanner.WritableMediaFileService.createOrUpdateChild(WritableMediaFileService.java:229) ~[!/:114.1.1]
jpsonic  |      at com.tesshu.jpsonic.service.scanner.WritableMediaFileService.updateChildren(WritableMediaFileService.java:176) ~[!/:114.1.1]
jpsonic  |      at com.tesshu.jpsonic.service.scanner.WritableMediaFileService.getChildrenOf(WritableMediaFileService.java:243) ~[!/:114.1.1]
jpsonic  |      at com.tesshu.jpsonic.service.scanner.ScannerProcedureService.scanFile(ScannerProcedureService.java:306) ~[!/:114.1.1]
jpsonic  |      at com.tesshu.jpsonic.service.scanner.ScannerProcedureService.scanFile(ScannerProcedureService.java:310) ~[!/:114.1.1]
jpsonic  |      at com.tesshu.jpsonic.service.scanner.ScannerProcedureService.scanFile(ScannerProcedureService.java:310) ~[!/:114.1.1]
jpsonic  |      at com.tesshu.jpsonic.service.scanner.ScannerProcedureService.scanFile(ScannerProcedureService.java:310) ~[!/:114.1.1]
jpsonic  |      at com.tesshu.jpsonic.service.scanner.ScannerProcedureService.lambda$parseFileStructure$3(ScannerProcedureService.java:276) ~[!/:114.1.1]
jpsonic  |      at java.base/java.util.Optional.ifPresent(Unknown Source) ~[na:na]
jpsonic  |      at com.tesshu.jpsonic.service.scanner.ScannerProcedureService.lambda$parseFileStructure$4(ScannerProcedureService.java:276) ~[!/:114.1.1]
jpsonic  |      at java.base/java.util.ArrayList.forEach(Unknown Source) ~[na:na]
jpsonic  |      at com.tesshu.jpsonic.service.scanner.ScannerProcedureService.parseFileStructure(ScannerProcedureService.java:275) ~[!/:114.1.1]
jpsonic  |      at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source) ~[na:na]
jpsonic  |      at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
jpsonic  |      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:354) ~[spring-aop-6.1.6.jar!/:6.1.6]
jpsonic  |      at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:716) ~[spring-aop-6.1.6.jar!/:6.1.6]
jpsonic  |      at com.tesshu.jpsonic.service.scanner.ScannerProcedureService$$SpringCGLIB$$0.parseFileStructure(<generated>) ~[!/:114.1.1]
jpsonic  |      at com.tesshu.jpsonic.service.scanner.MediaScannerServiceImpl.doScanLibrary(MediaScannerServiceImpl.java:162) ~[!/:114.1.1]
jpsonic  |      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
jpsonic  |      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
jpsonic  |      at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]

Thanks a lot!

tesshucom commented 1 month ago

In Japan, there is a custom of taking a long vacation in early May. I've been away from programming for a while. I've been working on some obvious bugs since I came back. Please note that we will address this issue soon.

As the exception indicates, the numeric field is probably non-well-formed. (Editor output is not appropriate.) I understand this is a frustrating issue, but please understand that it is not the most urgent issue that makes the difference between life and death for this project(or for me).

It may seem like just a few lines of change. However, it is unreasonable to think that we are actively working on improvements to tolerate unformed data, and that if something happens in the future, it will all be resolved here. There are more variations of format incompatibility problems than users may think. Alternatively, these improvement ideas should originally be discussed in the tag editor's project. It is the most important tag editor's responsibility to read all formats, good and bad, and output well-format data with specific specifications. That's the most important feature.

We conduct tests using data that comprehensively outputs fields from Music Center and Itunes, data inherited from Airsonic (general tag editor), data used in Jaudiotagger, etc. Neither of these assumes non-well-formed data. (Moreover, no such data existed.) Because it can't normally exist.

We recommend using highly reliable software for ripping from CDs and editing tag data. The combination of strict software and strict software is strict. Please note that non-warranted software comes with a statement that results are not guaranteed, but you should have seen it.　It's more constructive to avoid using poor quality editors.

If I may add. . . We wish you could. There are other ways to think of solutions other than requesting first aid. From my point of view, it would be helpful if you not only provided the specific incident, but also the name and version of the tag editor used for the output. (If you can no longer understand them and are trying to shift the blame to other software for the process behind it, you need to stop and think.) Or information regarding follow-up measures, such as whether the situation was improved by re-editing with other software. And in the very long term, that information is more useful. For other users too. OSS is a place to bring solutions together.

Tak-MK commented 1 month ago

First of all, hope your holidays were nice!

I'd like to say that I only added my comment out of willingness to help; I know it's not urgent, and even less in OSS projects. I'm saying this because you sounded upset from my perspective and I wanted to clarify that, at least from me, I'm not blaming anyone nor putting any pressure to no one... I'd be glad to make PRs myself if I knew Java but it's not the case, the most I can do is to share any bug, or things that look like a bug, for the maintainers of the project to solve if they consider and when they consider (as I've done in the past in this same project).

Having said that, I'm tagging my flac files using metaflac:

$ metaflac --version
metaflac 1.4.2

So I'm filling the flac comments manually, and probably that CDDB extra comment that I use for organization purposes in my server is the culprit since it's probably too long to fit in a field in the DB (except that now that you pointed it out, DATE is null so there's the issue, sorry for not looking into it before, I went directly to the CDDB comment...).

As for a "solution", I don't think tolerating malformed data is good, but I do think logging out whenever malformed data appears would be helpful to know where the problem comes from.

Thanks a lot!

P.S.: When the scan process hangs after that scan fail, it doesn't recover until I restart the process/docker, just as an extra info.

tesshucom commented 1 month ago

Thank you.

There is a reason why we don't just look at the errors.

P.S.: When the scan process hangs after that scan fail, it doesn't recover until I restart the process/docker, just as an extra info.

Please think of this as a data protective measure.

This is not bad behavior. That's the intended behavior. For example, in the case of other similar servers, a completely different process may overwrite it with a completely different value after the first write encountered a problem. It is possible to design a system that continues to operate without any problems, but this may lead to secondary failures or make it more difficult to understand the problem.

If a problem occurs with the scan, all major writing functions, such as Podacst and tag updates, will be locked. The current problem is that it is a bit too strict for users with non-well-formatted data. (This is not a problem for other users.) They may be gradually eased through issues like this.

Note that even after this format mitigation is introduced, the specifications of the exception and locking mechanisms will not change. Although this is unlikely in normal use, a similar lock would occur if memory overflowed or the network was disconnected.

Tak-MK commented 1 month ago

I agree with not having to check all the patterns and fields, but maybe outputting to the log which file is going to be scanned next would be helpful, if it’s not too much trouble. That way, I would know where to look at instead of guessing.

And thanks for the explanation regarding the crashing behavior.

Thanks!

2024年5月20日(月) 19:37 tesshu.com @.***>:

Thank you.

There is a reason why we don't just look at the errors.

  • The number of well-formatted patterns is finite.
  • The number of incorrect data file patterns is almost infinite. (Because it is not limited to field specifications)
  • Although not guaranteed... If we are looking at issues specific to a particular piece of software, the scale will be limited. Reproduction by third parties is also possible. We will be more likely to respond appropriately to the issue you raise.
    • If necessary, you can also validate other fields that are being processed by the editor you are having problems with.

P.S.: When the scan process hangs after that scan fail, it doesn't recover until I restart the process/docker, just as an extra info.

Please think of this as a data protective measure.

This is not bad behavior. That's the intended behavior. For example, in the case of other similar servers, a completely different process may overwrite it with a completely different value after the first write encountered a problem. It is possible to design a system that continues to operate without any problems, but this may lead to secondary failures or make it more difficult to understand the problem.

If a problem occurs with the scan, all major writing functions, such as Podacst and tag updates, will be locked. The current problem is that it is a bit too strict for users with non-well-formatted data. (This is not a problem for other users.) They may be gradually eased through issues like this.

Note that even after this format mitigation is introduced, the specifications of the exception and locking mechanisms will not change. Although this is unlikely in normal use, a similar lock would occur if memory overflowed or the network was disconnected.

— Reply to this email directly, view it on GitHub https://github.com/tesshucom/jpsonic/issues/2635#issuecomment-2120900707, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPMBZR5M2NHPTZP7MYTSQDZDIYFRAVCNFSM6AAAAABHB4O6K6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRQHEYDANZQG4 . You are receiving this because you commented.Message ID: @.***>

tesshucom commented 1 month ago

I agree with not having to check all the patterns and fields, but maybe outputting to the log which file is going to be scanned next would be helpful, if it’s not too much trouble.

Although this is not acceptable, why not try changing the log level first? Does that give you any hint? The topic of this issue is, to begin with, a topic that is separate from normal operations. Try using something like INFO DEBUG TRACE.

Jpsonic will never, ever use logs to navigate the processing flow during a scan. It was deliberately removed over time. This means that unnecessary disk access is reduced. Also, don't have expectations about the order in which files are processed. In extreme terms, such flow a tracking methods break down when processing is parallelized.

Tak-MK commented 1 month ago

I think I’ve stated it in my previous post, but just in case for clarification, I’ve used TRACE as log level and it didn’t help regarding to know what happened. I’ve used both INFO and DEBUG before that too but they obviously outputted less info.

I never asked for the logs being a source of truth for jpsonic, as they’re not meant for that, but what’s the point of not outputting useful info as the name of the file being processed to know why did it crashed?

I also get that the scanning is parallelized so I don’t expect any specific order, just a:

Log entry: “I’m going to scan X file” Log entry+1 (or +1/2/3 if it’s parallelized): “scan crashed”

2024年5月21日(火) 19:25 tesshu.com @.***>:

I agree with not having to check all the patterns and fields, but maybe outputting to the log which file is going to be scanned next would be helpful, if it’s not too much trouble.

Although this is not acceptable, why not try changing the log level first? Does that give you any hint? The topic of this issue is, to begin with, a topic that is separate from normal operations. Try using something like INFO DEBUG TRACE.

Jpsonic will never, ever use logs to navigate the processing flow during a scan. It was deliberately removed over time. This means that unnecessary disk access is reduced. Also, don't have expectations about the order in which files are processed. In extreme terms, such flow a tracking methods break down when processing is parallelized.

— Reply to this email directly, view it on GitHub https://github.com/tesshucom/jpsonic/issues/2635#issuecomment-2123100036, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPMBZWUJQKTQOHYW7WYOW3ZDN7QDAVCNFSM6AAAAABHB4O6K6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRTGEYDAMBTGY . You are receiving this because you commented.Message ID: @.***>

tesshucom commented 1 month ago

First, let's be clear that Jpsonic was not created to find non-well-formatted data in libraries, and that will never be its primary goal.　The reason why the behavior when non-standard data is read is postponed is because it is YAGNI. You Aren't Going to Need it. In other words, it's best not to add features until you actually need them.

Currently it is possible to analyze from a few dozen to over 100 data per second, but this upper limit will likely be raised in the future. If anything, the direction of interest is pointing in that direction.

Although it is not posted on the Wiki, my site has a hierarchy of data that is used as a model.

What this means is that the data formats of companies that provide music apps on multiple platforms and devices, or sell music resources including tags, are used as reference. In other words, we are referring to data formats that are thought to be widely distributed around the world. Of course, there are many cases where OSS software has a quality comparable to these.

Unfortunately, this issue deals with cases where none of these apply, but please note that such users are probably not the majority. Also, please note that assuming such a case and taking measures in advance is in the opposite direction from the main concern of Jpsonic. We could also call it an edge case or dirty when compared to the range of data I'm presenting.

The main work that would have been going on during this time will be stopped and I may invest my time and effort here. The code will be modified, test cases added, and speed verification performed. If possible, I would appreciate it if you could understand my way of thinking and Jpsonic's policy.

Also, remember that the software you choose to write is more important than the software you choose to read. For important processing, we recommend choosing reliable software over irregular software. Please reconfirm that if you use irregular software, you should do so at your own risk. In my opinion, the root cause lies in the process before Jpsonic processing.

Tak-MK commented 1 month ago

Thanks for the explanation!

I’d like to say that I’m glad that there’s a project to keep *sonic software alive, and since I’m choosing yours as the best one, I obviously respect your path of design and development.

So I’m just giving some advices and I trust in your judgement of taking them or not.

2024年5月21日(火) 20:28 tesshu.com @.***>:

First, let's be clear that Jpsonic was not created to find non-well-formatted data in libraries, and that will never be its primary goal. The reason why the behavior when non-standard data is read is postponed is because it is YAGNI. You Aren't Going to Need it. In other words, it's best not to add features until you actually need them.

Currently it is possible to analyze from a few dozen to over 100 data per second, but this upper limit will likely be raised in the future. If anything, the direction of interest is pointing in that direction.

Although it is not posted on the Wiki, my site has a hierarchy of data that is used as a model.

-

  1. A music management app for Windows released by Microsoft and Sony.

  2. A music management app for Windows released by Apple.

  3. Music data purchased from a major online music sales service (for reference only).

What this means is that the data formats of companies that provide music apps on multiple platforms and devices, or sell music resources including tags, are used as reference. In other words, we are referring to data formats that are thought to be widely distributed around the world. Of course, there are many cases where OSS software has a quality comparable to these.

Unfortunately, this issue deals with cases where none of these apply, but please note that such users are probably not the majority. Also, please note that assuming such a case and taking measures in advance is in the opposite direction from the main concern of Jpsonic. We could also call it an edge case or dirty when compared to the range of data I'm presenting.

The main work that would have been going on during this time will be stopped and I may invest my time and effort here. The code will be modified, test cases added, and speed verification performed. If possible, I would appreciate it if you could understand my way of thinking and Jpsonic's policy.

Also, remember that the software you choose to write is more important than the software you choose to read. For important processing, we recommend choosing reliable software over irregular software. Please reconfirm that if you use irregular software, you should do so at your own risk. In my opinion, the root cause lies in the process before Jpsonic processing.

— Reply to this email directly, view it on GitHub https://github.com/tesshucom/jpsonic/issues/2635#issuecomment-2123200797, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPMBZXP2G6NZ552BJIOM5LZDOG3PAVCNFSM6AAAAABHB4O6K6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRTGIYDANZZG4 . You are receiving this because you commented.Message ID: @.***>

tesshucom commented 1 month ago

Assuming you understand, let's put this issue on hold. It may be fixed, but the timing is not guaranteed. The reasons have already been mostly stated.

If I may add. .

postscript

If you would like a more accurate validation tool, please report any problematic data you identify, including the song format and tag version, and we may take it into consideration.