Any plan of stable release?

[yzlinaz]

This is great work and I like the idea of pure web-based JPIP interface a lot. I am trying to adopt this webjpip for my project, I am wondering if there is any recent plan of a stable release ? As I noted the last commit is about a year ago.

Thanks

[MaMazav]

I don't have a clear roadmap right now, although I'm back to the development work on the last month. (I'm in the middle of a huge refactoring, current development is done on https://github.com/MaMazav/ImageDecoderFramework.js - a generic framework library for heavy image decoding, like JPIP).

I can tag the current time point as "version 0.1", just to have a versioning management :) from my short experience it's working well now, but notice that no real QA was done.

Anyway I would be very happy to hear suggestion about what should be done to stabilize the library. Including bugs/issues, design/API comments (which I guess there are a lot) and active contribution to the library of any kind. Anyway notice that I'm working on it on a free time basis, so I can't promise anything.

[yzlinaz]

Hi, Thanks for the response and I apologize for my late reply. This webjpip.js project works for me. Thought there is a few issues that I am currently trying to tackle. The most outstanding issue is the decoding speed. On my computer with a 4k display, using JPIPLeafletDemo to decode a ~4k x 2k region takes 6,7 seconds with 5 parallel web workers. In comparison, to use the jpx.js to decode a similar size image with single thread, takes only 6 or 7 seconds. Theoretically, the parallel decoding should shorten time to 1 to 2 seconds. I looked at the code, the reason seems to be that the asynchronous decoding of spatial tiles, though parallelized, involves a lot of repetitive decoding of the commonly shared code blocks between tiles.

Now I am trying to change the code so that the parallelization is done in the codeblocks level, and the codeblocks in the display region is only decoded once. But I am wondering during development, have you thought about this ?

P.S. I have found several other issues and some bug fix, will try to contribute them in different thread.

Thanks,

[MaMazav]

Very interesting. I've done a lot of profiling to achieve satisfyable results, but they focused on special use cases, mainly for viewer.

First, there is a simple way to check your suggestion: The JpxJpipImage can accept tile size in its options argument. Choose a tile size equal your image and it will be decoded as a whole.

There may be another causes. I saw that a main performance degregation caused by the data copied between the threads. I tackled this problem but did not finish all problems here. If this is the problem then replacing any WorkerProxy classes by its original proxied class will solve the problem (but will stuck the UI. This only for check). The replacement should be easy, I hope I'm not wrong.

If your suggestion is correct a deep design change should be made in both jpx.js and webjpip.js (and apparently image-decoder-framework.js) to allow reuse of decoded data. I heard (but didn't check it) that most of decode time is arithmetic decoding. It may allow to share decode of arithmetic coded data before wavelet decode and pixels calculation.

If this is not confidential proprietary I'll be happy to see your concrete image and how you use the webjpip.js to try tackle the problem.

[yzlinaz]

Hi, Thanks for you timely reply. So here is an example of a jp2 file that slows down the viewer. It would be great if you could look at the image and provide your thoughts on possible reason that is slow. jp2s.zip

I haven't looked at the suggestion of replacing WorkerProxy classs with orignal proxied class, will let when I have time to try that. But I can be quite certain that the bottleneck is the arithmetic decoder, and that's the reason I proposed my solution.

Thanks,

[MaMazav]

I still didn't debug your images but I thought about another possible reason: By default decoding image is done progressively by quality layers - without reuse of older decoded data. Maybe cancelling the progressive decoding will increase the performance.

[MaMazav]

Did you get a region of 4k x 2k to be fully decoded? I couldn't make it work on your image. I tried to decode the region 0,0-4kx2k and some of the tiles were not decoded (in the first and second tile columns). In addition when tried to watch your image in a viewer I got some exception when zooming in. Did you check the decoding in Debugging mode? If you had exceptions while decoding it may decrease performance significantly.

I'm afraid that this image exposes more serious bug in webjpip.js. It seems to be a wrong calculation in tile partitioning into precincts. Until this bug is solved, we can check performance on an image which behaves well: precinct and tile sizes of power of two, and each tile should be composed of only full tiles.

[yzlinaz]

Yes, there was some wrong calculation of partitions into precincts, I fixed at my end, and it works for me now.

The fix that needs to be done are in "webjpip.js/jpipcore/imagestructures/jpiptilestructure.js", change line 177-197 to var precinctX = 0; var precinctY = 0; //add these if (precinctsInCodestreamPartPerLevelPerComponent !== null) { var firstPrecinctsRange = precinctsInCodestreamPartPerLevelPerComponent[0][0]; precinctX = firstPrecinctsRange.minPrecinctX; precinctY = firstPrecinctsRange.minPrecinctY; }

    // A.6.1 in part 1: Core Coding System
    var componentStructure = componentStructures[0];
     var precinctsX = componentStructure.getNumPrecinctsX(0);
    var precinctsY = componentStructure.getNumPrecinctsY(0);
    var precinctIndexInCompRes = precinctX + precinctY * precinctsX;
    var setableIterator = {
        precinctIndexInComponentResolution: precinctIndexInCompRes, /*

was set to 0*/ component: 0, precinctX: precinctX, precinctY: precinctY, resolutionLevel: 0, isInCodestreamPart: true };

I am sure there is cleaner way to write these as this is just a quick fix for me. But will works for this image nonetheless.

On Sun, Jul 3, 2016 at 11:54 AM, MaMazav notifications@github.com wrote:

Did you get a region of 4k x 2k to be fully decoded? I couldn't make it work on your image. I tried to decode the region 0,0-4kx2k and some of the tiles were not decoded (in the first and second tile columns). In addition when tried to watch your image in a viewer I got some exception when zooming in. Did you check the decoding in Debugging mode? If you had exceptions while decoding it may decrease performance significantly.

I'm afraid that this image exposes more serious bug in webjpip.js. It seems to be a wrong calculation in tile partitioning into precincts. Until this bug is solved, we can check performance on an image which behaves well: precinct and tile sizes of power of two, and each tile should be composed of only full tiles.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/MaMazav/webjpip.js/issues/3#issuecomment-230169039, or mute the thread https://github.com/notifications/unsubscribe/AQzsJl-hDFVILjYXmkDbvGV4JgRhRkz2ks5qSAVIgaJpZM4IddvV .

[MaMazav]

I'll look at it. It takes much longer than 7 seconds in my laptop (didn't compare to jpx alone).

I prettified your suggestion, you may review that in the following change: https://github.com/MaMazav/webjpip.js/commit/15554de375898850617bb5768e2d21448e1fac2e#diff-a177a0ab4a0502d9ca77a45c6e9659bd

If you would like to you may open a PR with this change and I'll take it.

[yzlinaz]

Thanks for the change. I recently found a new source of inefficiency that may partly explain this problem. The image is sent you is encoded with a a tile size of {8192,8192}. If you were to use a smaller tile size during encoding,e.g. {512,512}, the decoding/displaying time would be significantly reduced. I suspect there is something very inefficient when handling code-stream that has large tile-size, because, if everything was done correctly the tile-size in code stream should not affect decoding time so significantly.

On Tue, Jul 5, 2016 at 2:01 AM, MaMazav notifications@github.com wrote:

I'll look at it. It takes much longer than 7 seconds in my laptop (didn't compare to jpx alone).

I prettified your suggestion, you may review that in the following change: 15554de https://github.com/MaMazav/webjpip.js/commit/15554de375898850617bb5768e2d21448e1fac2e

If you would like to you may open a PR with this change and I'll take it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/MaMazav/webjpip.js/issues/3#issuecomment-230424506, or mute the thread https://github.com/notifications/unsubscribe/AQzsJl0RXxWvO07vdx4XAOkF1wd4AyVeks5qSh10gaJpZM4IddvV .

[MaMazav]

Unless there is a bug, the library knows to decode only data in the region of interest. It creates a codestream and data with empty holes for data which is out of region to avoid redundant decoding. It is limitted only by precinct size, so the interesting question is what is the precinct size (or is there a bug in the region of interest mechanism).

[MaMazav]

Finally I reached the root cause of that factor, and you are correct - it caused by the fact that decoding workers perform same decoding some times. However it is caused by a logical problem rather than a bug or silly inefficiency in the code (explanation below). A full solution to the problem is to allow sharing of decoding. I'm working now on integrating the imageDecoderFramework.js and webjpip.js library, when I finish that I consider to add another phase of common decoding to imageDecoderFramework.js, which will enable webjpip.js to predecode arithmetic coded data. Workarounds that make the situation a little better until the full solution is implemented: all workarounds deal with changing logical tile size of the image. The logical tile size only controls the size of the units that are fetched and decoded and does not require changing the j2k file. This is very easy to do that - just add tileWidth and tileHeight properties to the options object of the JpxJpipImage constructor. However the performance improvement I measured is not significant.

1. Change tileWidth and tileHeight to the sizes of the area you decode (4096x2048). This option is the best if you want to get a decoded data ASAP and do not care progressive rendering and not having different sizes of decode requests. It performs well if requests are aligned to this size.
2. If you cannot do the previous solution but still want to improve the situation a little, change tileWidth and tileHeight to be a number which does not fit exactly the real precinct sizes, e.g. 270 instead of 256. Sounds strange? read the explanation below.

I guess that you may achieve reasonable results if you adapt the parameter to your use-case according to above workaround guidelines. For example logical tile size of 600x600 gives reasonable results which still support progressiveness.

EXPLANATION: In an ideal world we could use only the wavelet coefficients needed for our region. That is, if we are decoding region of size 256x256 we needs 128x128x3 coefficients of the best level, 64x64x3 of the next level and so on, to a sum of 256x256. However there are two problems with that:

• The problem I guess you already understood - we cannot use part of tile. We need to arithmetic decode all tile. Thus we need 128x128x3 coefficients of each level.
• Another serious problem is that wavelet uses all neighbor pixels to decompress precinct (to a maximum margin of 4 pixels), to get smooth image. It means that when the logical tiles lies exactly on an edge of a precinct - it needs also its neighbor precinct for decode. It means that we need 128x128x12 (x12 instead of x3) precincts at the best resolution level for decoding of most of the precincts! and part of other levels also affected similary... That's why I suggested to choose logical tile size of 270 instead of 256.

If you calculate the effect of those things you will get factor of about 6, which may explain the performance degradation you saw.

(b.t.w. even without the wavelet-inherent problems, in my laptop there is no linear speedup for the decoding task although embarrassedly parallel task and 4 cores).

[MaMazav]

A lot of time has passed but finally I found some time to try to optimize. I had very limited success... I'm documenting the conclusions of last profiling (after optimizations) so can be continued one day (most probably by me). Except of saving work by reusing already decoded coefficients I found some more inefficiencies, e.g. parsing the same codestream headers more than once and some other js problems like massive use of setTimeout, bind, etc.

• The baseline for most optimized decoding is pure pdf.js' jpx decoder. It takes about 10 seconds on my machine to decode the example file.
• On the other hand, it takes 45 seconds to achieve same result with the webjpip.js full flow.
• Cancelling the web-workers and calling jpx on main thread seems to reduce several seconds.
• Changing image-decoder-framework.js' applicative tile size to be all image (so the pixel-decoding part is done on all the image at once, and no cropping of buffer needed in jpx) reduces the decoding to 25-30 seconds.
• Taking the baseline (original pdf.js's jpx which takes 10 seconds to decode) and just replacing the jpx decoder with webjpip.js' revised one (which revised to support partial decoding, but without using the partial decoding feature) increases decoding time from 10 seconds to 15 seconds.
• Also profiling the baseline vs. the non-web-worker with applicative tile size for all image showed that the jpx.js decoding part is the most heave part in both, specifically the pixel-decoding parts which includes the wavelets functions as iterate, filter, transform (rather than coefficient arithmetic decoding, which are next in the list of heavy functions but only after pixel-decoding ones). Comparing smaller parts of jpx.js functions consistently shows that the the new version takes some tens of percents more in time (e.g. 7 seconds for iterate in new version vs. 5 seconds in baseline version).

It turns out that the highest potential for optimizations is the pixel-decoding part in pdf.js' jpx, i.e. the wavelet optimizations. Diffing with the non-revised functions show that the main changes are, as expected, the calculation of region of interest. I can think of three reasons for problems there -

1. Using multiplications and divisions to project the region to other levels/tiles.
2. Using objects to pack the region which introduce level of indirection and redundant memory management.
3. Copying coefficients from one array to other (done for some reasons, e.g. cutting exact region of interest, and copying coefficients from previous stage of decode to exact position in array). Profiling showed that it less than expected - about 700ms.

[anthonycarbone]

Hi, Would you have a simple html demo of some sort that works with the current state of webjpip.js? I see that the initial commit did but since the refactor it doesn't work with the new version. Also, how would you describe the current state of webjpip.js? In its current state is it capable of simple viewing of JPIP images? Thanks.

[MaMazav]

Working demo over Leaflet can be found here: http://mamazav.github.io/webjpip.js/

Similarly, this demo is based on image-decoder-framework.js interface, so it can be used also with Cesium or adapted to other viewers as well, as described here: http://mamazav.github.io/image-decoder-framework.js/

Regards stability - I used the library on some heavy images and it worked pretty good. However it was never used in real customer facing scenario. Also notice that there are j2k features that are not supported, as described in readme.md.