Moving some functionality to the server.

[GoogleCodeExporter]

May i suggest move some funcionality to the Server?

1) Torrent Download.
2) LevenshteinDistance at server level. 
A complete populated list of anime (GetAllAnime), takes 5 seconds, from the 
server to the desktop client, in the same machine. (Yea, i know i have a lot of 
anime), When i finish figure out NHibernate. Maybe i can help you to optimize 
serializations timings, or create some kind of cache, does nHibernate have a 
memory-cache?)

Original issue reported on code.google.com by maximo.p...@gmail.com on 22 Aug 2012 at 3:34

[GoogleCodeExporter]

1. yeah I had thought about that because of code duplication, but not high 
priority at the moment

2. same

3. yeah it can be pretty slow. NHibernate doesn't cache collections.
   however I have some of my own caching in there, take a look at the StatCache class 

Original comment by werndly...@gmail.com on 22 Aug 2012 at 4:54

[GoogleCodeExporter]

Okey, i did some testing on my setup. First i was thinking the more sluggish 
part was serialization/deserialization and packet transmission. But it's not 
the case. Seems the main resource hog is the ORM (nHibernate).

The AnimeSeries GetAll took 4-5 seconds, all the time. MyAnime took from 6 to 
18 seconds to refresh any group Filter page.

I did a comparasion against the entity framework and load and serialize the 
entire anime list.

AnimeSeries GetAll, Took 640 ms the first time, 12 ms the second time and 12 ms 
the third time.

Seems the entity framework, have some kind of caching mechanism. 

I going to test is the results are similar with SQLLite and MySQL.

Original comment by maximo.p...@gmail.com on 23 Aug 2012 at 3:53

[GoogleCodeExporter]

Which database are you using?

I am using MSSQL, getting the ALL group filter in MA3 is pretty mich instant

Original comment by werndly...@gmail.com on 23 Aug 2012 at 4:52

[GoogleCodeExporter]

MSSQL,  2778 series. I have 11 Tb of anime... My Anime 2 was almost instant
againt SQLlite. (~ amount of anime)

Original comment by maximo.p...@gmail.com on 23 Aug 2012 at 5:58

[GoogleCodeExporter]

ah that really is a lot of anime, I only have 1100

MSSQL is faster than mysql, and 10x faster than sqlite

Original comment by werndly...@gmail.com on 23 Aug 2012 at 7:32

[GoogleCodeExporter]

Okey, Leo, i took my time, and recoded the database backend to Entity Framework 
5. To my dismay performance was on par o below par. :(  More on this later.

Then i figured it out. The major hog in the Server is the GroupFilter 
calculation.
So i took the liberty of adding GroupFilter Caching to StatCache. Now, My Anime 
3 refresh is below a second on any group or filter. 

On my machine, "Got groups for filter EVAL" took from 6500 ms to 9500 ms. 
Now it tooks less than 1 ms. More Than 6000X the perfomance :)

Attached is the server (Warning, i put the lastest AutomaticUpdaterWPF.dll, 
older versions didn't compile right with me, older version hangs this 
JMMServer.exe)
.Compiled with Visual Studio 2012.

Source code is available and not much is changed. (statCache, 
JMMImplementation, and 5 or 6 Entities on the Save/Delete methods). 

I have two branches one is playground with EF 5. 

And the other with this update.

If you want, i can branch yours in here but i'll need access. Also, since i 
fxxed up, with the Japanese Sxxt on My Anime 2, and i don't want to change the 
curse of JMM, since is your child. I'll prefer that you merge the needed on the 
main branch.

Note on EF 5, lots of optimization can be done with Lambdas on the current 
code, mixing code with the database queries. Since a lot of code logic can be 
passed into DB queries.

I also identified other thing that can be optimized, but need some 
architectural changes.

Client queries are large and consuming. With Some clever caching in the server, 
and little changes in the large contracts, it's only need to send the deltas, 
the changes (adds, updates or deletes) of the contracts. Less bandwidth used, 
less time, small packets, less serialization done by the server.

Also if you upgrade WCF to .NET 4.5. You can use WCF compression. Not tested 
but may get some extra juice.

Original comment by maximo.p...@gmail.com on 29 Aug 2012 at 5:42

Attachments:

• JMMServer_WithGroupFilterCaching.rar

[GoogleCodeExporter]

Thanks a lot maximo, bit busy with work for the next week or so.
But what you have done sounds really good, and I will merge in soon :)

FYI the reason I have tried to avoid direct sql as much as possible, so that 
everythinn would work across all types of databases

Original comment by werndly...@gmail.com on 29 Aug 2012 at 7:12

[GoogleCodeExporter]

Oh no source code in the attachment.
Let me know if you want to merge it in yourself or let me do it.

Original comment by werndly...@gmail.com on 29 Aug 2012 at 7:15

[GoogleCodeExporter]

Here are the sources, on the mixed queries/code, you don't use Entity Framework 
with SQL sentences, you use lambda expressions, and lambda expressions are part 
of C#. There is no sql, and you don't need to code sql sentences. Lambda are 
very powerfull, you can mix and match, queries, with your normal methods. 
Lambdas can be applied to collections, can be List<>,IEnumerables, etc, can be 
XML sources, or can be SQL sources.

Per example GetMostRecentlyAdded in AnimeSeriesRepository.

Original comment by maximo.p...@gmail.com on 29 Aug 2012 at 4:20

Attachments:

• GroupFilterPatch.rar
• ServerEF5.rar

[GoogleCodeExporter]

I have applied and tested the changes in GroupFilterPatch.rar, awesome work.
I'll run at home for a week do now.

I'll look at ServerEF5.rar soon.

Original comment by werndly...@gmail.com on 7 Sep 2012 at 2:33

[GoogleCodeExporter]

I assume Issue #107 has the more recent version of the AnimeSeries.cs changes?

Original comment by werndly...@gmail.com on 7 Sep 2012 at 2:37

[GoogleCodeExporter]

Yes :)

Original comment by maximo.p...@gmail.com on 7 Sep 2012 at 2:40

[GoogleCodeExporter]

Hi, I have a few thoughts to add regarding the data layer/NHibernate, etc.

I've noticed that you're micro managing the session lifetime of NHibernate, ie. 
A new session is being opened for every operation being performed. By doing 
this you're not getting any use out of the first level cache (ie. if you're 
performing an operation which ends up selecting the same entity more than once, 
then you'll be making a trip to the database each time instead of just once).

I attached the Management Studio Profiler to the SQL Server database I was 
testing with and found that a lot  of queries were being performed (far more 
than I would expect). In certain scenarios I was seeing the same query (with 
the same parameters) being executed one or two hundred times.
A quick workaround for this would be to enable the NHibernate second level 
cache and adjust mappings/queries as required to take advantage of it. I played 
around with this myself and found a dramatic decrease in the number of queries 
being executed.

However, I really think it would be a good idea to move away from the 
repository pattern that is being used and instead switch to something like a 
service layer that just uses the session directly (The ISession is basically 
the repository in this case).

Also, I'd re-work the models/mappings to better take advantage of NHibernate. 
For example the property AniDB_Anime.Tags uses AniDB_TagRepository to get the 
list of tags. I'm sure you're aware that you can map the relationships with 
NHibernate and have it do all the work for you (including cascading of deletes 
and batch loading), etc. (This of course would require lazy loading which can't 
be taken advantage of when micro managing the session).

Hopefully most of that made sense. I'd be happy to provide any assistance/code 
if you need any help.

Original comment by mclei...@gmail.com on 11 Oct 2012 at 9:40

[GoogleCodeExporter]

Hi and thanks for your feedback.

1. The reason for this is because early on I ran into problems when I used a 
globally shared session. I can't remember the details but I think it was only 
when using the sqlite database. So I ended up using the current method.
However I was aware of the performance issues with the current method and in 
one or two places I ended up over-riding this.
You can see this in the CreateCategories method in AniDB_Anime.cs

If you have any ideas here, let me know.
Otherwise I need to go back and try it out again to see what my issue was 
previously

2. Any links on how I use the second level cache. Or if you tried it against 
the source code, maybe you could send me what you modified so I can see how it 
works.

3. Is it possible to still use the repositories, and share a global session?

Original comment by werndly...@gmail.com on 11 Oct 2012 at 11:47

[GoogleCodeExporter]

Also feel free to contact me by email or on the IRC channel

Original comment by werndly...@gmail.com on 11 Oct 2012 at 11:47

[GoogleCodeExporter]

Original comment by werndly...@gmail.com on 19 Aug 2014 at 2:54

• Changed state: Fixed