Newsletter not listing new episodes added for existing shows

Stryk3r78 commented 7 years ago

Ombi Version: 2.1.1538

Update Branch: Early Access Preview

Media Sever: Plex

Media Server Version: 1.4.3.3433

Operating System: Windows 10

Problem Description: Since the recent changes to the newsletter the new tv episodes section is now not listing any new episodes added from existing shows, but is only listing new episodes for shows that were newly added. Since new tv shows are not added regularly, it would really be nice if it could have all new eps for new and existing shows listed that were recently added. It used to do this fine before the recent changes.

Reproduction Steps: Sending a test newsletter after adding a new episode for an existing show will not include the episode in the newsletter list. But if I add a new show, then add new episodes, those will show up in the newsletter.

Ombi-robot commented 7 years ago

Hi! Thanks for the issue report. Before a real human comes by, please make sure you used our bug report format. Before posting make sure you also read our FAQ and known issues. Make the title describe your issue. Having "not working" or "I get this bug" for 100 issues, isn't really helpful. If we need more information or there is some progress we tag the issue or update the tag and keep you updated. Cheers! Ombi Support Team

tidusjar commented 7 years ago

Do you have episode searching on?

Stryk3r78 commented 7 years ago

No it's off, I've never turned it on as I don't have any episodes in my Plex library that aren't in Sonarr. Like I mentioned it picks up newly added episodes for new shows (for 1 week), but doesn't pick up new eps for any existing shows now (including new shows 1 week after they are added) and it did before with episode searching off as well.

Is this a requirement now to make this work?

tidusjar commented 7 years ago

For Ombi to add episodes to the newsletter then we require episode searching.

Can you upgrade to the latest EAP version and I just pushed some fixes to the newsletter around this.

Stryk3r78 commented 7 years ago

ok I have enabled episode searching and updated to EAP version 1554. Will send another test newsletter tomorrow to see if new eps are being added to the newsletter.

Thanks for your help.

Stryk3r78 commented 7 years ago

Still seems like a no go, even after enabling episode searching and upgrading to the latest EAP it's still not working. 3 new movies and 3 new episodes for existing shows were downloaded last night, and only the movies are showing in the newsletter.

tidusjar commented 7 years ago

Are there any errors in the log? It's difficult to diagnose an issue without knowing what has happened.

When did the episode cacher last run? Was it before the newsletter sent?

On Sat, 25 Mar 2017, 1:23 pm Stryk3r78, notifications@github.com wrote:

Still seems like a no go, even after enabling episode searching and upgrading to the latest EAP it's still not working. 3 new movies and 3 new episodes for existing shows were downloaded last night, and only the movies are showing in the newsletter.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/tidusjar/Ombi/issues/1264#issuecomment-289211413, or mute the thread https://github.com/notifications/unsubscribe-auth/AGVaLMIOHu_5Y2uYuiNmAAwFn9S2Lgehks5rpRVrgaJpZM4MgmRN .

Stryk3r78 commented 7 years ago

no errors in the logs, i manually ran the episode cacher last night after I enabled episode searching. I've run it again now even though it should run every 12 hours. It seems to complete very quickly though it says "success" immediately, whereas some other jobs take a few seconds to complete, but it's not leaving anything in the logs.

I'll give my server a full reboot and see if it makes any difference. The problem is difficult to trace as everything seems to be working and doesn't create any errors, it just doesn't seem to pickup that new episodes are added unless I've just added the show to Sonarr for the first time. then it works fine.

If there is something more I can look at to assist you please let me know, I'm just not sure what else to try?

JDIacobbo commented 7 years ago

I am also having this issue. Enable Episode Searching is on in the Plex settings. I have attached a copy of the log at Trace Level.

Application Version: 2.1.1554 OS: Windows 8/Windows Server 2012 System Version: 4.0.30319.42000 Branch: Early Access Preview

Job Name Last Run
Plex Episode Cacher Sun, 26 Mar 2017 15:25:38 GMT
Database Backup Sun, 26 Mar 2017 15:22:46 GMT
Request Limit Reset Sun, 26 Mar 2017 15:27:45 GMT
Database Cleanup Sun, 26 Mar 2017 15:44:37 GMT
Sonarr Cacher Sun, 26 Mar 2017 15:56:08 GMT
CouchPotato Cacher Mon, 06 Mar 2017 20:12:24 GMT
Plex Availability Cacher Sun, 26 Mar 2017 15:58:08 GMT
Plex Cacher Sun, 26 Mar 2017 15:53:34 GMT
Plex User Checker Sun, 26 Mar 2017 15:54:09 GMT
Watcher Cacher Sun, 26 Mar 2017 15:55:08 GMT
Request Fault Queue Handler Sun, 26 Mar 2017 15:15:45 GMT
Recently Added Email Notification Fri, 24 Mar 2017 11:00:04 GMT
Radarr Cacher Sun, 26 Mar 2017 15:53:11 GMT

Ombi_Newsletter_Log.txt

tidusjar commented 7 years ago

So the episode searcher even if manually triggered will not run until 11 hours has passed since the last run (due to with Plex it is very resource consuming).

Can you guys open up your databased (Ombi.sqlite) with something like SQLiteBrowser and look at the PlexEpisodeContent table to see if there is anything in there?

JDIacobbo commented 7 years ago

So the episode searcher even if manually triggered will not run until 11 hours has passed since the last run (due to with Plex it is very resource consuming).

Can you guys open up your databased (Ombi.sqlite) with something like SQLiteBrowser and look at the PlexEpisodeContent table to see if there is anything in there?

I checked the PlexEpisodes table of PlexRequests.sqlite and there is definitely content there (11618 items!). I tried again to run the newsletter, but this time I am generating a blank newsletter and nothing is getting emailed out.

tidusjar commented 7 years ago

It's possible that it's marked things as already sent.

Maybe try and clear out some stuff from the RecentlyAddedLog table?

JDIacobbo commented 7 years ago

It's possible that it's marked things as already sent.

Maybe try and clear out some stuff from the RecentlyAddedLog table?

Just updated to the most recent EAP (2.1.1564). After the update the newsletter is generating again, but new episodes aren't listed (restarting the application probably fixed the sending issue). All I get for TV is a new series that was recently added, listed.

Just to be clear, under Settings --> Plex --> Enable Episode Searching is checked.

tidusjar commented 7 years ago

Maybe try and clear out the RecentlyAddedLog table?

Then run it again?

Stryk3r78 commented 7 years ago

my PlexEpisode table is blank, but I see my existing shows as well as my movies listed in the PlexContent table, I don't have a PlexEpisodeContent table though.

I upgraded to the latest EAP, unchecked episode searching, re-checked episode searching and restarted OMBI, and it says it's "indexing my tv shows and movies" but the episode table is still blank. Still no errors in the logs.

JDIacobbo commented 7 years ago

Ran DELETE FROM RECENTLYADDEDLOG
Cleared Logs and set to Debug.
Restarted the Ombi service.
Ran test newsletter to admin.
Received e-mail.

So I think we may have an improvement here. After clearing the recently added log and rerunning the newsletter, EVERYTHING was listed on the test newsletter. I'm assuming this will be resolved the next time the does its compare to the last (side note: is there a way to force this?). But listed under the TV series I am now seeing:

Season: 1, Episode: 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,

Season: 2, Episode: 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,

Season: 3, Episode: 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,

Season: 4, Episode: 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,

tidusjar commented 7 years ago

Excellent.

So yes it will show everything because Ombi thinks that this is the very first time it has run. The next newsletter will will compare what is in your library to what has been sent, if there is stuff that hasn't been sent then we will send it.

JDIacobbo commented 7 years ago

Excellent.

So yes it will show everything because Ombi thinks that this is the very first time it has run. The next newsletter will will compare what is in your library to what has been sent, if there is stuff that hasn't been sent then we will send it.

Is there a way to force the compare before the next scheduled run? I don't want to send my users the same email that almost broke my inbox!

tidusjar commented 7 years ago

It will do the compare when the schedule runs. It's part of the job :)

CyberMew commented 7 years ago

I'm also not getting newly added episodes. The title text is there in the email but the contents are blank. Can we actually make the test newsletter useful by actually including the contents instead of an empty email? Send it but not mark the contents as sent.

Stryk3r78 commented 7 years ago

My PlexEpisodes table is still empty, what can I do to fix this?

tidusjar commented 7 years ago

@Stryk3r78 It's probably because your either have episode searching disabled Or there is an error when trying to get the Plex Episodes. Can you find that error and raise a new issue?

Stryk3r78 commented 7 years ago

Bu this was my original issue? Why did you close it? it's not resolved?

There are no errors and episode searching is Enabled. like I said I even turned it off and back on.

tidusjar commented 7 years ago

Sorry I must have misread it.

If there is no Episodes in your PlexEpisodes table then there is a problem with getting the episodes out of Plex. There would be some sort of error for this.

What is your log level set to?

Stryk3r78 commented 7 years ago

Log level is set to Error. And this is all I see in there.

Begin executing GetProfiles call to Radarr for quality profiles Ombi.Core.Setup Info 3/29/2017 8:31:29 AM Begin executing GetProfiles call to Sonarr for quality profiles Ombi.Core.Setup Info 3/29/2017 8:31:29 AM Begin executing GetProfiles call to Sonarr for quality profiles Ombi.Core.Setup Info 3/29/2017 8:30:19 AM Begin executing GetProfiles call to Radarr for quality profiles Ombi.Core.Setup Info 3/29/2017 8:30:19 AM

Stryk3r78 commented 7 years ago

I set it to Trace, and now the logs are filling up with this:

Message Area Log Level Date Request made to http://192.168.2.99:32400/library/metadata/19029/children with status code OK. The response was Ombi.Api.ApiRequest Trace 3/30/2017 8:26:45 AM Request made to http://192.168.2.99:32400/library/metadata/19029 with status code OK. The response was Ombi.Api.ApiRequest Trace 3/30/2017 8:26:45 AM Request made to http://192.168.2.99:32400/library/metadata/65699/children with status code OK. The response was Ombi.Api.ApiRequest Trace 3/30/2017 8:26:45 AM Request made to http://192.168.2.99:32400/library/metadata/65699 with status code OK. The response was Ombi.Api.ApiRequest Trace 3/30/2017 8:26:45 AM Request made to http://192.168.2.99:32400/library/metadata/64404/children with status code OK. The response was Ombi.Api.ApiRequest Trace 3/30/2017 8:26:45 AM Request made to http://192.168.2.99:32400/library/metadata/64404 with status code OK. The response was Ombi.Api.ApiRequest Trace 3/30/2017 8:26:45 AM Request made to http://192.168.2.99:32400/library/metadata/2355/children with status code OK. The response was Ombi.Api.ApiRequest Trace 3/30/2017 8:26:45 AM Request made to http://192.168.2.99:32400/library/metadata/2355 with status code OK. The response was Ombi.Api.ApiRequest Trace 3/30/2017 8:26:45 AM

seems this is normal, no?

Stryk3r78 commented 7 years ago

set to debug, shows this:

Message Area Log Level Date Done albums Ombi.Services.Jobs.PlexContentCacher Debug 3/30/2017 8:26:50 AM Done TV Ombi.Services.Jobs.PlexContentCacher Debug 3/30/2017 8:26:50 AM Done movies Ombi.Services.Jobs.PlexContentCacher Debug 3/30/2017 8:26:47 AM done all that, moving onto the DB now Ombi.Services.Jobs.PlexContentCacher Debug 3/30/2017 8:26:45 AM

tidusjar commented 7 years ago

Everything seems normal.

Can you open your database (with Ombi closed) and execute the following in SQLiteBrowser: DROP TABLE PlexEpisodes Press save then restart Ombi and wait for the Job to run (it will not run unless 11 hours have passed)

Stryk3r78 commented 7 years ago

Done

Query executed successfully: DROP TABLE PlexEpisodes (took 1ms)

I restarted OMBI and it has recreated the table. What should I leave the error level at to see any problems? Debug? Error? Trace?

tidusjar commented 7 years ago

Just put it to error :)

On Thu, 30 Mar 2017 at 13:34 Stryk3r78 notifications@github.com wrote:

Done

Query executed successfully: DROP TABLE PlexEpisodes (took 1ms)

I restarted OMBI and it has recreated the table. What should I leave the error level at to see any problems? Debug? Error? Trace?

— You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub https://github.com/tidusjar/Ombi/issues/1264#issuecomment-290397000, or mute the thread https://github.com/notifications/unsubscribe-auth/AGVaLK2Vu4X_AYotVDGef76UcH2n7fw_ks5rq6FfgaJpZM4MgmRN .

Stryk3r78 commented 7 years ago

ok thanks,

I there any way to force the job to run sooner? I have a fairly powerful server so I'm not too concerned about the resources but would like to not have to wait 11 hrs to see if this is working every time I make a change?

tidusjar commented 7 years ago

Not currently. It's hard coded in to make sure we don't constantly run it. Nowa'days we don't really need that check. I'll remove it in a future version.

On Thu, 30 Mar 2017 at 13:43 Stryk3r78 notifications@github.com wrote:

ok thanks,

I there any way to force the job to run sooner? I have a fairly powerful server so I'm not too concerned about the resources but would like to not have to wait 11 hrs to see if this is working every time I make a change?

— You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub https://github.com/tidusjar/Ombi/issues/1264#issuecomment-290399039, or mute the thread https://github.com/notifications/unsubscribe-auth/AGVaLA60imTbxclo5O80KmFCYnipq8K2ks5rq6N5gaJpZM4MgmRN .

Stryk3r78 commented 7 years ago

So the weekly newsletter went out again with no tv shows episodes listed and my Plex Episodes table is still blank. No errors in the logs since I updated to to the new EAP yesterday at 5pm.

Message Area Log Level Date Begin executing GetProfiles call to Sonarr for quality profiles Ombi.Core.Setup Info 3/30/2017 4:59:44 PM Begin executing GetProfiles call to Radarr for quality profiles Ombi.Core.Setup Info 3/30/2017 4:59:44 PM

tidusjar commented 7 years ago

Great. There is clearly a problem trying to get the episodes from Plex. If the table was populated then the newsletter wouldn't have an issue.

There must be an error in your log somewhere, if it fails it will always log an error.

Stryk3r78 commented 7 years ago

Also seemed someone else opened a ticket for a similar if not the same issue #1297. It definitely seems to be a problem with populating the episodes table, as no matter what I do it is not adding any data into it. I have upgraded to the latest Plexpass server Version 1.5.2.3557 as well, so not sure if they've changed anything but I would think it should be giving me an error of some kind? But there no errors? I pasted exactly what is listed. Do I need to use a different log level than error?

Stryk3r78 commented 7 years ago

shows it ran the episode caching job at around 4am my time. and as there is nothing in the logs since I started the service at 5pm yesterday.

Job Name Last Run
Plex Cacher Fri, 31 Mar 2017 10:59:55 GMT
Plex User Checker Fri, 31 Mar 2017 11:48:46 GMT
Watcher Cacher Fri, 31 Mar 2017 11:01:45 GMT
Plex Availability Cacher Fri, 31 Mar 2017 11:49:45 GMT
Plex Episode Cacher Fri, 31 Mar 2017 09:09:45 GMT
CouchPotato Cacher Mon, 01 Jan 0001 00:00:00 GMT
Request Fault Queue Handler Fri, 31 Mar 2017 09:12:45 GMT
Database Backup Thu, 30 Mar 2017 21:19:45 GMT
Request Limit Reset Fri, 31 Mar 2017 09:24:45 GMT
Database Cleanup Thu, 30 Mar 2017 21:34:45 GMT
Radarr Cacher Fri, 31 Mar 2017 10:59:47 GMT
Sonarr Cacher Fri, 31 Mar 2017 11:02:46 GMT
Recently Added Email Notification Fri, 31 Mar 2017 11:00:04 GMT

Shiroihige commented 7 years ago

Im also having this issue. I have episode searching turned on. Running Ombi v2.2.1591, Plex v1.5.2.3557.

No episodes get added to newsletter. If I run the episode cacher manually, it doesn't add anything to the PlexEpisodes table. With log level set to error, nothing is written to logs during any point of this manual run.

The PlexContent table does however contain all the movies and tv series that are in the library.

JDIacobbo commented 7 years ago

So a week after clearing out the PlexEpisodes and RecentlyAddedLog table, I am not receiving episodes in the newsletter. The first newsletter that ran on Friday was blank. I added a movie and several new episodes, but the test email is only showing the movie. New episodes are not showing up.

jablesR commented 7 years ago

in my experience, New shows get added to the list, but new episodes of existing shows do not get added.

Stryk3r78 commented 7 years ago

@jablesR that is exactly what this issues says in title and description. Also there are no errors in the logs to help track down the problem it seems, and there are about 3 issues opened right now reporting the same problem.

jablesR commented 7 years ago

@Stryk3r78 just trying to add to the discussion, new to this github thing. hoping we can help with the solution here as the newsletter is a great addition to ombi.

jjlambkin commented 7 years ago

@jablesR I agree. That would be nice to have. I dont think however that that is how the application is intended to behave. I could be wrong though. I am pretty sure I saw something along these lines in the feature suggestions area.

Shiroihige commented 7 years ago

I've been having this issue and just upgraded to the EAP to test another issue that was being fixed. After upgrading the newsletter now included new shows that had been added and movies, but does not include individual episodes that been added. So it seems to be an issue with how ombi finds individual episodes that are being added or something in that process chain.

Stryk3r78 commented 7 years ago

I appreciate the confirmation that I'm not crazy or alone in this situation. I'm trying to provide more info for the devs but there isn't much to be able to provide as everything "works" fine, and new things are added fine but existing shows don't add.

@jjlambkin it used to work this way, they changed the way the newsletter works and this has been the result. before they used to just query the API and dump all teh results, this worked fine but ended up with duplicate results. Then Plex changed the API in a fairly recent plexpass version and the OMBI dev had to update to compensate as it was sending BLANK newsletters for a while.

Once they fixed that, ever since it has been checking for dupes and not sending them. but it seems it's not scraping the existing shows and now only added NEW eps for NEW shows. whereas before it was sending all new shows from the recently added TV api.

This is definetly not how it should work, as bascially now the tv section is empty as you don't always add a new TV show every week.

GilbN commented 7 years ago

@stryk3r78 yes. Looks to me that the tv show part of the newsletter works the same as the movie part. It processes a TV show as a movie and doesn't add it on a per episode basis but per show.

Stryk3r78 commented 7 years ago

@Shiroihige the issue you are describing is the one I've had all along since the newsletter API scraping changes. It has always added newly added shows fine for me, just nothing from existing shows hence my issue title.

My plex episodes table in blank even with episode searching turned on, but this doesn't seem to stop it from recognizing new show episodes. but alas, as soon as a week passes and that shows isn't "new" anymore any new eps downloaded are once again ignored and not listed. I must download about 20-30 new eps a week and nothing has been listed for about 3 weeks and counting whereas all my new movies are listed (and no longer adding dupes from previous weeks which I believe was part of teh changes that seem to be causing this new problem). I believe it's no longer scraping teh API and instead looking to the local plex episodes DB to find episodes and then comparing them against the recently added log DB table. Since the plex episodes table is blank.... it think nothing new is added and send the tv section blank,

But it doesn't log any error about not populating the table, and the job runs every 11 hours without issue from what I can tell. So I don't know what else to do??

Stryk3r78 commented 7 years ago

@gilbN I concur, if I download a "new show" and add 5 episodes for it, it still only lists the show in the newsletter as 1 item, not 5 episodes like it should. And then the following week, it's not a "new show" anymore so it's not added even if new episodes are added.

but this still doesn't explxain why the epsiode table doesn't populate when it runs the episode scraper which is what I believe is the other part of the issue since as long as it's not recording a list of episodes it never knows what new or not.

GilbN commented 7 years ago

@stryk3r78 hmm. well it does list the episodes I have downloaded on new shows for me. http://m.imgur.com/UgWFYND

Stryk3r78 commented 7 years ago

@gilbN that must be new, mine didn't show that episode list that last time it actually posted a new show on March 24th.

Just a MASSIVE cover photo and this text underneath.

Marvel's Iron Fist 2017

Genre: Drama, Action, Crime After his family meets a tragic fate while on expedition in China, a young Rand is adopted by the people of the mystical lost city of K'un-Lun, where he's taught a magical fighting style. Years later, he returns to New York to fight crime.

ofurpesi commented 7 years ago

Got this when i ran the newsletter job:

System.NullReferenceException: Object reference not set to an instance of an object. at Ombi.Services.Jobs.RecentlyAddedNewsletter.PlexRecentlyAddedNewsletter.GenerateTvHtml(IEnumerable`1 recenetlyAddedTv, StringBuilder sb) in C:\projects\requestplex\Ombi.Services\Jobs\RecentlyAddedNewsletter\PlexRecentlyAddedNewsletter.cs:line 334

Still not showing new episodes in the newsletter

PS. Running EAP

Ombi-app / Ombi

Newsletter not listing new episodes added for existing shows #1264