Enhanced Template Engine

[JoeBiellik]

I've been looking at implementing a much more advanced templating system in v3. Currently, in a number of places, we let users use a basic template system using predefined variable interpolation, e.g. {ShowName} - S{Season:2}E{Episode}[-E{Episode2}] - {EpisodeName}. I've seen a few issues that revolve around the limitations of this system and I think in v3 we should improve it.

I've played around with Scriban which is a more advanced implementation of Liquid templates and it works really well. The Liquid syntax is very similar to how we currently do interpolation but allows full scripting if desired by more advanced users. A Liquid equivalent of the template above would be something like {{show.name}} - S{{season.number | pad}}E{{episode.number | pad}} - {{episode.name}}. The main benefit to this is we can provide the user with modifier functions like pad, lowercase etc. There are a number of other popular templating systems we could use but they all seem to have a very similar syntax.

This change would impact "filename templates" (#59), "custom search" (#278) and could be used for more powerful configuration of things like the season folder name or even a full default show location, e.g.: D:\TV\{{show.name}}\Season {{season.number | pad}}\{{show.name}} - S{{season.number | pad}}E{{episode.number | pad}} - {{episode.name}}

This could also be used to completely replace the current DownloadIdentifiers with a much more powerful system that lets the user entirely define where they want metadata, quick example:

This would cover all the current media center files and we could add presets for them. All of these options could be set globally then overridden per show (#29).

Thoughts?

[SirSparkles]

Hey. Think this is a great plan. Just want to make sure there are quick fire ways to get the current core types added. I’d see us have an ‘Add Kodi images’ which quickly pre-adds the 3-4 types of images needed. Further I like the idea is using the same template for the season / show folders. Useful to define some shows with s01e01 style to be in season folders and other daily shows split by year/month.

Overall i’m keen as long as we have shortcuts to keep it easy to use (I did shortcuts to automate even the checkboxes we currently have) and provide a translation from the current setup to the new one. If we make it too technical then we’ll just have loads of people in the forums asking for help!

[JoeBiellik]

I agree maintaining ease of use for current users is important, what do you think of the following if a user picked the Kodi preset?

[SirSparkles]

I like it, but I image this being an 'advanced' screen. The basic one could just be a few radio buttons / dropdown for Kodi/Plex etc, then the user can select 'Advanced..' if they want to configure this complexity.

Maybe add 'source' as a column to be customised later?

For single folder shows I'm assuming you'd have show.location = season.location = episode.location? Rather than having to customise these settings. However, how would we deal with contention in this case? (ie the rules above would try and copy both the season poster and show poster to the same file) Just something to be worked through.

Shrinking of Mede8er images - how would you deal with that? New image types? 'Shrunk Season Poster'? or maybe new format 'Shrunk JPEG'?

Just a few thoughts from the top of my head - looks really good though.

[JoeBiellik]

Yes, the form can be in the preferences behind an advanced button. There could be a "source" or "preset" column but that might be confusing if a user later edits an item.

Depending on how we handle a "directory structure template", files could be anywhere relative to the other directories so good path handling is important. Collisions are possible so we can either try to limit adding multiple conflicting options when configuring the settings or handle it at scan time. I've got this mostly implemented in v3, so I'll do more testing.

Rather than having a file type for resized images, I'd like to just add resizing options to all image files. Option to scale/crop to a set size, maintaining/forcing the aspect ratio. This also gives users the option to set their own resizing as desired.

[Andy-ABTec]

This is all looking really interesting, can't wait to play...

You're probably both aware of the Kodi formats already, but just out of interest I exported a TV Show from Kodi. I see the following (bearing in mind I'm setup to use posters): -

In the show folder: banner.jpg, fanart.jpg, poster.jpg, season01-poster.jpg etc and season-all-banner.jpg, tvshow.nfo

In a season folder: an xxx.nfo file and an xxx-thumb.jpg where xxx matches the video file name ie showName-SyyEzz-episodeName

[JoeBiellik]

@Andy-ABTec Kodi supports many names/formats for the files and will search through a list until it finds one, so the current settings should work. That said the files you list are its internal defaults and probably what we should default the Kodi preset to.

I'll get this working and committed up so you can have a look.

[Andy-ABTec]

Great, please dont think I'm sticking my nose in, just didn't want you guys to miss something...

[JoeBiellik]

I've added these features to my pull request and would very much like any feedback on how the template system works.

You should be able to set the templates in the preferences, configure the "Media Center Files", add a show and scan. Currently scanning only lists all files defined by the templates but it should be enough for testing.

@Andy-ABTec Is it easier if I upload a build for you to test? Wasn't sure if you had the compiler setup and would appreciate the feedback.

[SirSparkles]

OK, thanks will take a look. I'm looking forward to taking a look at a few of the v3 things that I have not had the chance to take a look at yet. I'll try and take a look tonight, but if I don't manage to then it maybe a week or so (sorry - prior commitments). Press on though - I'm sure you're heading in a good direction.

I assume that a backup of preference files etc is wise.

[JoeBiellik]

@MarkSummerville No problem. v3 writes to .json files in %appdata%\TV Rename so you shouldn't have collisions with v2 files.

[SirSparkles]

Hi @JoeBiellik Took a look last night and it looks really good. App is much zippier (difficult to tell if that's due to improved processing or the reduction in some of the baggage though!) I like the cleaner RHS too. Show lookups on TVDB seemed slower, but I have not tested properly - just an observation.

One observation (maybe it only applies to me) is that I do not have a common root for all shows. They are split by type (kids shows, docos, normal etc). So having x:/TV/ as a common root causes me problems. We have the concept of many folders for the 'find shows' function. Maybe we allow many and then select from the locations when adding a show.

I still see the custom image download UI above being behind an 'Advanced...' dialog, but I like it and it will give us great flexibility.

How do you want to progress? I don't want you to think you have to do this all on your own, but at the same time I don't want to get in the way.

[JoeBiellik]

By slower show lookups, do you mean when searching to add a new show or the first full download of that show? The full show download can definitely be optimized - its downloading everything.

The default show location option doesn't need to be kept if you think most users have shows in multiple locations. I mainly added it so that I didn't have to do too much typing when debugging multiple shows, but I'm now working on a better "folder monitor", as well as being able to drag a folder into the app to prefill the "add show" dialog. Perhaps the add show form could search the saved paths from the folder monitor for prefilling.

A lot of the UI is just thrown together for testing right now, I'm more interested in abstracting the key functionality before rethinking the forms.

I'm not really sure the best way to proceed; right now I'm just trying to get my vague ideas implemented. Any help is always appreciated but I realize I'm making massive modifications purely based on my opinion without any warning, which is why I think it might be best for me to get my changes out and then keep whats valuable.

It shouldn't take too much longer for me to port all the existing scan functionality over to the v3 branch, and at that point, I think we do a full review of the code and decide which of my modifications are worth keeping. We can properly document the code, write unit tests, and then finally port the changes back into master? I don't mean to change the direction of the project but rather offer alternative code as an option.

On the subject of the template engine, so far I've happy with it, has anyone found any issues?

[Andy-ABTec]

Sorry @JoeBiellik just spotted this (been playing with this newfangled template)... I don't have a compiler setup (and probably never will) but I'd love to see, can you zip up and mail an exe that I can drop into my tvrename folder? Ta muchley...

[SirSparkles]

I was referring to lookup from the text name of a show to get the list of shows matching that string. I don't see that you've changed that much there. Maybe just a delay as it connected? I wouldn't worry about it.

I think probably most people will have a common root, but given that one of the attraction of TV Rename is its flexibility then I think allowing a choice from the 'folder monitor' folders would be better. If there is only one 'folder monitor' folder then it auto chooses that folder

I'm happy bumbling along with minor bug fixes while you get the bulk of the code refactored. It feels like the wholescale refactoring is a one person job due to the size/impact of the changes planned. Once you have a version that has say 80% of the changes you envisage then I'll start getting up to speed on the way it works and help get it ready for beta. Also I'm relaxed if some of the UI changes do/do not come into v3 - doesn't bother me either way.

I have not really tested the template engine - more just a high level review to see the usablility. I'm sure I can tell from the code, but is there a list of the tags/formats that the template can use? I'll start doing some more detailed testing on it. One thing I noticed is that you can multi-add a kodi preset (leading to double up) and add inconsistent rules (ie different files/formats going to the same location/filename)

[Andy-ABTec]

I'll second the multi-add Kodi, + the show search throws an exception if you click it with a blank entry

[JoeBiellik]

@MarkSummerville The idea is that during scanning we use ProcessedXXX models, e.g. ProcessedShow which contains everything Show has and a few bits specific to the local files. Most of the templates should use these models, but there are a few places we have to process them against the basic models as we haven't scanned yet. I'm still playing around with the exact relationships between the models. Anything exposed in TemplateHelpers is available in the templates as helper functions.

What do you guys suggest for duplicate files/presets? Bearing in mind the new form would be behind the existing media center preferences and is for advanced users, I think its okay to allow duplicates as long as it's handled correctly during a scan.

I've pushed up changes which let you actually process the metadata files and generate them. I'm trying to follow the latest Kodi/PyTivo/Mede8er formats, so I would appreciate any testing and comparison between v3, v2 and official software output.

Edit: Worth pointing out that the metadata file generation isn't complete, for example, it doesn't yet build Kodi episode NFOs correctly.

[SirSparkles]

Hi, I noticed that the episodes are not in order should be date, episode number. Currently just by date, so if 2 episodes are on the same day then they are sometimes in the wrong order.

Re uniqueness I'd keep location/filename unique. if someone tries to add a preset with the same location&filename as an existing then we'd alert them that we're overriding an existing row. We'd still get clashes that we'd have to handle in code (as there are multiple ways of specifying the same path)

[JoeBiellik]

@MarkSummerville By out of order do you mean the scan results? I've not ported over the column sorters yet because I'm playing around with having them user sortable as well.

[SirSparkles]

no - under 'my shows' when you click on a season and see the list of episodes on the RHS

[JoeBiellik]

Thanks, my latest code should fix a number of ordering issues.