Decision on content addition

[manami-project]

Current Situation

Content of the database

The database contains anime which do not fit into one of the following categories:

• commercials/promotions
• stop motion productions
• music videos
• pure CG productions
• abstract animations
• unrelated shorts
• any non-japanese productions (korean, chinese, american...)
• anime before 1970

The exclusion of specific titles is tool-supported, but not completely automated.

Benefits

• The database is subjectively clean.
• The database is smaller in size than it would be if contained other not fitting entries.

Problems

• There is a lot of manual intervention necessary in the process.
  • This takes time and some anime might not be added to the database for a while.
• There are always anime which should be excluded, but which are not identified.
  • That means that the database will never be as clean as it's supposed to be.

Merging

The base for all entries is always myanimelist.net (MAL). Links from other meta data provider are merged into these existig entries. That means that the meta data are also based off the entry from MAL.

Problems

• Over the past few months MAL has been down. This has proven to be a real problem, because
  • no new anime had been added during this time. That means that missing titles which were available on other meta data provider were not available in this database.
  • Very, VERY few changes/updates had been made on MAL during that time. So the database was outdated.
• Meta data provider might provide different meta data. E.g. Both MAL and another meta data provider lists the same special with 2 episodes. Both are merges in one entry in this database. Another episodes comes out. MAL changes the existing entry and lists the anime now with 3 episodes whereas the other meta data provider creates a new entry. That would mean the following: The entry in this DB lists the link from the other meta data provider with data which is not correct for this meta data provider. And the entry of the other meta data provider containting the third episode will never be merged.
• Anime entries which cannot be merged itno an existing MAl entry will never appear in the database. E.g. if MAL lists an anime as OVA and another meta data provider lists the same anime as special the link for the other meta data provider will never appear in the database.

Possible change

A possible solution to address the first problem would be to remove all rules to filter anime. To get rid of the second problem it would be best to stop being based on MAL. Instead of the current logic, a hit-first approach should be established in merge process. It is important to adapt both changes or none, becauses if you solely want to stop being based on MAL, but still maintain a subjectively clean database you drastically increase the manual effort for filtering anime, because you have to filter an anime for each meta data provider.

Functionality

• An anime is added to the database if it does not already exists no matter which meta data provider it originated from
• entries which already exists will be merged based on the current logic
• All anime of a meta data provider will be added to the database. No exclusion will be applied.

Your Feedback

For me it makes totally sense to adapt these changes in populating the database. However I would love to hear from anyone interested in this project why you are interested in it. Do you like the apprach of a clean database based on the filter categories mentioned above or do you just like that there is an anime offline database and would you therefore prefer a database that strives for completeness? Please leave a comment on this topic on this issue.

[ghost]

I'm really interested in this project as I'm looking to build an iOS Anime Tracker and don't want to tie it down to other services. I really love the work you're doing! Can I ask why you're MAL-based? Are the other meta data providers not as comprehensive? I'd really be willing to help the best I can!

[ghost]

Also do you mind explaining the not-found.json file further?

[manami-project]

Hey @faizanabbas, sorry to reply this late. First of all thank you very much for your Feedback. I'm glad you like this project and I hope it will help you to create a very cool app.

The reason for being based on MAL is that this database project is created out of the need for offline caching in my other project manami. I created manami solely for my own needs. Because I linked my animes based on MAL both manami and this database initially only supported MAL. When I started to add additional meta data provider it was convenient to be based on MAL, because MAL still has the biggest database, I used MAL myself and I wouldn't have to think about any additional merge logic. Long story short: It just made things easier for me. I realized that interest in this project is higher than expected and I want to provide titles from anidb, anilist and kitsu even if my automated process is not able to merge them.

The not-found.json file: All IDs that lead to dead entries are listed in this file. Dead entries may occur if a meta data provider deletes duplicates or decides that an entry should not be listed due to their rules. I personally use this file in manami to let manami know that it doesn't have to look for the title itself using an http request, because nothing will come out of it. The file is also part of the automated process which creates this database file. Maybe the file should be renamed to dead-entries.json to make it's purpose more clear?

[ghost]

Thank you! I see now!

Maybe the file should be renamed to dead-entries.json to make it's purpose more clear?

Yeah I think it should; it's a little more explicit.