search

Carnaux / NFT-Marker-Creator

This editor creates NFT markers for ARTOOKIT 5.x
https://carnaux.github.io/NFT-Marker-Creator/
334 stars 162 forks source link

About jpeg and future development of NFT marker creator #6

Closed kalwalt closed 4 years ago

kalwalt commented 5 years ago

We continue here the discussion started on my PR for NFT https://github.com/kalwalt/jsartoolkit5/pull/2#issuecomment-534261520 @Carnaux we can leave here our thoughts and solutions for the problem.

Carnaux commented 5 years ago

To generate the marker files is needed some information about the image:

The function "readImageFromFile" gets all the information to set a struct and then other functions can generate the marker files.

Using an HTML input gives only the Uint8 array, the width, and height. I tried setting a placeholder DPI, but it didn't work. I was trying to generate a new jpeg file and store it in the MEMFS and then read it "locally" with the GenTexData.c, so I wouldn't need to change the original code from the unity artoolkit.

Maybe the solution is easy, but I didn't have much time to work on it :disappointed:

kalwalt commented 5 years ago

Thank you @Carnaux i will look into it when i have a bit of time.

Carnaux commented 5 years ago

@ThorstenBux Can you help me understand better the marker generation?

ThorstenBux commented 5 years ago

I can try :). What are your questions?

Carnaux commented 5 years ago

I solved most of my questions working today :smiley:, I'm generating the iset and fset files, but they don't work, they have drastically different file size than the generated by the genTexData.exe.

nicolocarpignoli commented 5 years ago

well, that's a great progress :) good luck, as soon as I will have time I will help you

kalwalt commented 5 years ago

I solved most of my questions working today , I'm generating the iset and fset files, but they don't work, they have drastically different file size than the generated by the genTexData.exe.

That really strange @Carnaux but as said Nicolo a great progress!

Carnaux commented 4 years ago

Just to document one of my attempts:

I tried to execute the genTexData.exe directly in node.js. If you don't pass a file as parameter the expected error message appears, so it runs and does "console.log", but when I pass the path for the file it won't show any sign that it is running.

I think that the .exe is waiting for a DPI input and is getting the code stuck. I tested with a simple hello world .exe compiled from c++ that just outputs and the code worked, when I added an input it got stuck. A possible solution is the use of child_process.spawn().

If the input problem gets solved, the data got by EXIF.js can create default values for the inputs of the .exe.

What the current code can do:

nicolocarpignoli commented 4 years ago

@Carnaux I think that if you can come up with specific questions and maybe an online example to try (not mandatory) you can ask to Thorsten. He's very will to help. Just come up with some questions here and I will see to make it happen.

Carnaux commented 4 years ago

Update: I posted about an error dpi_num, but it was a typo error, and it was solved.

I think I figured out what was the deal. The way I was passing the pixel data as a parameter was wrong, solving that made the generation almost equal to the genTexData.exe, all logs are being shown, and all generation steps are seen to be working. But I noticed that the files still different sizes so they don't work with the NFT example, I believe it is the ARUint8 pixel array data structure, the feature extraction doesn't generate the correct values. @ThorstenBux, I saw in a forum that the ARUint8 pixel structure changes depending on the SO, it apply in this case too?

PS: Now all the steps of the marker generation work, but it takes 25 minutes to finish. The repository was updated, so anyone can test the generation times.

kalwalt commented 4 years ago

I will look at for the ARUint8 issue. 25 minutes It Is a huge time...

Carnaux commented 4 years ago

I developed a node app too, that uses the same markerCreator.min.js as the browser version(the current one), just to test generation times.

The results:

genTexData.exe: 3 min Node app: 13 min Browser version: 25 min

Some differences:

genTexData.exe:

Node app:

The values of filtered features are all slightly higher, but it doesn't log feature coordinates for high DPI values, only for 75 or below. Currently, it is faster than the browser, but I don't know how it will behave when it is hosted.

Browser:

The extracted and filtered values aren't correct, but it logs all coordinates for all DPIs.

nicolocarpignoli commented 4 years ago

Having wrong/few features could be the cause for the non-recognisation on Web.

nicolocarpignoli commented 4 years ago

@Carnaux

I think that our 'first' aim was to make it work with Web technologies, but maybe this computation is too heavy for the browser. In my opinion, it will be great to make it work on node, for example. It will be JS code and runnable on EVERY machine, windows or linux or mac. So it will be a great improvement from current state-of-the-art, that works only on windows.

I think we have to focus on make it work on node environment, and then, eventually in the future, port it also on the browsers.

But for a lot (maybe every) use case, to generate the descriptors 'offline', and then load them on the web, will be enough.

nicolocarpignoli commented 4 years ago

If you agree with me, you can focus your efforts on experiments/debug with node version only, like you did on your last post

kalwalt commented 4 years ago

@Carnaux yes maybe a node version will be also fine and easier to develop. @nicolocarpignoli i can make my own Nft markers also under linux, but you need to build the artoolkitx project and this can be a little cumbersome to insall all the necessary dependencies, so in the scenario of a "normal" user it's a kind of blocking thing, better a lot an alternative app that can run on every OS...

Carnaux commented 4 years ago

Yeah, I agree. I'm building a new node app, and when it is finished I will reorganize the repository with the main app and other branch to the NftMarkerCreator.min.js build.

Carnaux commented 4 years ago

Done, the app is on the master branch, you guys can test it. But I still stuck at the feature extraction, I tested changing the order of the RGB pixel array, but still the same as I said, the node app only logs feature coordinates for low dpi values.

Carnaux commented 4 years ago

Finally!!!! With this last commit, the code works, you now can generate custom NFT Markers!!!! 🎉 Please, test and report any errors or bugs! @kalwalt @nicolocarpignoli

ThorstenBux commented 4 years ago

I've tested and it is running with other jpg files as well. but I can't see any output files .fset .iset that are created. I've deleted the existing ones added a new jpeg to input. It runs with this output

[info] Name = [info] first = 70
[info] Filename: asa
[info] --
Generator started at 2019-12-13 09:15:42 +1300
[info] MAX_THRESH  = 0.900000
[info] MIN_THRESH  = 0.550000
[info] SD_THRESH   = 8.000000
[info] SURF_FEATURE = 100
[info]  min allow 3.733000.
[info] Image DPI (1): 3.733000
[info] Image DPI (2): 4.703285
[info] Image DPI (3): 5.925768
[info] Image DPI (4): 7.466000
[info] Image DPI (5): 9.406571
[info] Image DPI (6): 11.851537
[info] Image DPI (7): 14.932002
[info] Image DPI (8): 18.813144
[info] Image DPI (9): 23.703076
[info] Image DPI (10): 29.864005
[info] Image DPI (11): 37.626289
[info] Image DPI (12): 47.406155
[info] Image DPI (13): 59.728014
[info] Image DPI (14): 75.252583
[info] Image DPI (15): 94.812315
[info] Image DPI (16): 119.456034
[info] Image DPI (17): 150.000000
[info] Generating ImageSet...
[info]    (Source image xsize=1500, ysize=1125, channels=3, dpi=150.0).
[info]   Done.
[info] Saving to asa.iset...
[info]   Done.
[info] Generating FeatureList...
[info] Start for 150.000000 dpi image.
[info] p: 6575032
[info]          ImageSize = 1687500[pixel]
[info] Extracted features =   58037[pixel]
[info]  Filtered features =   33837[pixel]

But does not create output files

Carnaux commented 4 years ago

The process finishes? On my PC it took around 14 min to generate.

ThorstenBux commented 4 years ago

It looks like it gets stuck because of this does not contain embedded resolution data, and no resolution specified on command-line.

Carnaux commented 4 years ago

Weird, that log is in a function that should not be called. Can you send me the image that you are using?

ThorstenBux commented 4 years ago

Edited the jpeg now I get TypeError: Cannot read property '0' of undefined

ThorstenBux commented 4 years ago

Sorry that log is out of the c version of the tool :) I build that and ran it there.

ThorstenBux commented 4 years ago

Image on the way

ThorstenBux commented 4 years ago

https://drive.google.com/file/d/1cs-1n7njiJ4rnnrHTrNjyBolYvqxD58H/view?usp=sharing

ThorstenBux commented 4 years ago

(I hope google doesn't change the file)

ThorstenBux commented 4 years ago

Ok, got it now. The link above: That image fails for some reason I don't understand. I've reverted back to the previous image and waited longer it does work, it just takes a while.

Carnaux commented 4 years ago

It is the jpeg decoder, I will add this to a to-do list for a future update.

Edit: @ThorstenBux https://images.app.goo.gl/6JFAu3reU3bjzUKP6 this one works. I added the handling for empty EXIF info.

Edit2: It was generated in 5s!! 😮

Carnaux commented 4 years ago

One thing I realized: The images I was using for testing were all 1600 < in width and height, so they were big images. Small images are much faster to generate, but I believe that it has few details for the feature extraction.

kalwalt commented 4 years ago

@Carnaux @ThorstenBux I will test now!

kalwalt commented 4 years ago

:tada: Yesss, it works !! :tada: see the log

kalwalt commented 4 years ago

Tested with this my image https://www.kalwaltart.com/assets/images/uploads/cubist_dragon.jpg i used for my AR made with Artivive read this article

nicolocarpignoli commented 4 years ago

Hi!

I tried on macbook pro. It finishes in ~10-15seconds but no descriptors generated. I got those warnings (although they're on stderr, to me they look like errors):


stderr: /Users/nicolo/libjpeg-for-jsartoolkit5/jdhuff.c:454:13: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
  { 0, ((-1)<<1) + 1, ((-1)<<2) + 1, ((-1)<<3) + 1, ((-1)<<4) + 1,
        ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdhuff.c:454:28: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
  { 0, ((-1)<<1) + 1, ((-1)<<2) + 1, ((-1)<<3) + 1, ((-1)<<4) + 1,
                       ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdhuff.c:454:43: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
  { 0, ((-1)<<1) + 1, ((-1)<<2) + 1, ((-1)<<3) + 1, ((-1)<<4) + 1,
                                      ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdhuff.c:454:58: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
  { 0, ((-1)<<1) + 1, ((-1)<<2) + 1, ((-1)<<3) + 1, ((-1)<<4) + 1,
                                                     ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdhuff.c:455:10: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<5) + 1, ((-1)<<6) + 1, ((-1)<<7) + 1, ((-1)<<8) + 1,
     ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdhuff.c:455:25: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<5) + 1, ((-1)<<6) + 1, ((-1)<<7) + 1, ((-1)<<8) + 1,
                    ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdhuff.c:455:40: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<5) + 1, ((-1)<<6) + 1, ((-1)<<7) + 1, ((-1)<<8) + 1,
                                   ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdhuff.c:455:55: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<5) + 1, ((-1)<<6) + 1, ((-1)<<7) + 1, ((-1)<<8) + 1,
                                                  ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdhuff.c:456:10: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<9) + 1, ((-1)<<10) + 1, ((-1)<<11) + 1, ((-1)<<12) + 1,
     ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdhuff.c:456:25: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<9) + 1, ((-1)<<10) + 1, ((-1)<<11) + 1, ((-1)<<12) + 1,
                    ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdhuff.c:456:41: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<9) + 1, ((-1)<<10) + 1, ((-1)<<11) + 1, ((-1)<<12) + 1,
                                    ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdhuff.c:456:57: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<9) + 1, ((-1)<<10) + 1, ((-1)<<11) + 1, ((-1)<<12) + 1,
                                                    ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdhuff.c:457:10: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<13) + 1, ((-1)<<14) + 1, ((-1)<<15) + 1 };
     ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdhuff.c:457:26: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<13) + 1, ((-1)<<14) + 1, ((-1)<<15) + 1 };
                     ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdhuff.c:457:42: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<13) + 1, ((-1)<<14) + 1, ((-1)<<15) + 1 };
                                     ~~~~^
15 warnings generated.
/Users/nicolo/libjpeg-for-jsartoolkit5/jdphuff.c:214:13: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
  { 0, ((-1)<<1) + 1, ((-1)<<2) + 1, ((-1)<<3) + 1, ((-1)<<4) + 1,
        ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdphuff.c:214:28: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
  { 0, ((-1)<<1) + 1, ((-1)<<2) + 1, ((-1)<<3) + 1, ((-1)<<4) + 1,
                       ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdphuff.c:214:43: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
  { 0, ((-1)<<1) + 1, ((-1)<<2) + 1, ((-1)<<3) + 1, ((-1)<<4) + 1,
                                      ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdphuff.c:214:58: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
  { 0, ((-1)<<1) + 1, ((-1)<<2) + 1, ((-1)<<3) + 1, ((-1)<<4) + 1,
                                                     ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdphuff.c:215:10: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<5) + 1, ((-1)<<6) + 1, ((-1)<<7) + 1, ((-1)<<8) + 1,
     ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdphuff.c:215:25: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<5) + 1, ((-1)<<6) + 1, ((-1)<<7) + 1, ((-1)<<8) + 1,
                    ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdphuff.c:215:40: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<5) + 1, ((-1)<<6) + 1, ((-1)<<7) + 1, ((-1)<<8) + 1,
                                   ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdphuff.c:215:55: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<5) + 1, ((-1)<<6) + 1, ((-1)<<7) + 1, ((-1)<<8) + 1,
                                                  ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdphuff.c:216:10: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<9) + 1, ((-1)<<10) + 1, ((-1)<<11) + 1, ((-1)<<12) + 1,
     ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdphuff.c:216:25: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<9) + 1, ((-1)<<10) + 1, ((-1)<<11) + 1, ((-1)<<12) + 1,
                    ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdphuff.c:216:41: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<9) + 1, ((-1)<<10) + 1, ((-1)<<11) + 1, ((-1)<<12) + 1,
                                    ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdphuff.c:216:57: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<9) + 1, ((-1)<<10) + 1, ((-1)<<11) + 1, ((-1)<<12) + 1,
                                                    ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdphuff.c:217:10: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<13) + 1, ((-1)<<14) + 1, ((-1)<<15) + 1 };
     ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdphuff.c:217:26: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<13) + 1, ((-1)<<14) + 1, ((-1)<<15) + 1 };
                     ~~~~^
/Users/nicolo/libjpeg-for-jsartoolkit5/jdphuff.c:217:42: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    ((-1)<<13) + 1, ((-1)<<14) + 1, ((-1)<<15) + 1 };
                                     ~~~~^
15 warnings generated.

Running command: /usr/local/Cellar/emscripten/1.38.44/bin/emcc -I/Users/nicolo/artoolkit5/include -I/Users/nicolo/NFT-Marker-Creator/build/ -I/Users/nicolo/NFT-Marker-Creator/emscripten/ -I/Users/nicolo/artoolkit5/lib/SRC/KPM/FreakMatcher -I/Users/nicolo/libjpeg  /Users/nicolo/NFT-Marker-Creator/build/libar.bc /Users/nicolo/NFT-Marker-Creator/build/libjpeg.bc /Users/nicolo/NFT-Marker-Creator/emscripten/assemble.c -s EXPORTED_FUNCTIONS=["_createImageSet"] -s EXTRA_EXPORTED_RUNTIME_METHODS=["FS"]  -O3  -Wno-warn-absolute-paths  -s TOTAL_MEMORY=268435456  -s ALLOW_MEMORY_GROWTH=1  -s USE_ZLIB=1 -s ERROR_ON_UNDEFINED_SYMBOLS=0 -s ASSERTIONS=1 --memory-init-file 0  -s FORCE_FILESYSTEM=1 -s WASM=0   -D HAVE_NFT  -o /Users/nicolo/NFT-Marker-Creator/build/NftMarkerCreator.min.js 

stderr: shared:WARNING: not all asm.js optimizations are possible with ALLOW_MEMORY_GROWTH, disabling those. [-Walmost-asm]

Jobs completed

any idea?

nicolocarpignoli commented 4 years ago

Tried with thorsten image above

edit: following the readme, I was not able to do this point

4. Edit the .split(/\s+/).join(' PATH TO LIBJPEG') in the file makem.js.

Cannot find that line, maybe Readme is obsolete?

I'm doing this on dev branch.

nicolocarpignoli commented 4 years ago

which version of emscripten do you have guys? Mine is 1.38.44

Carnaux commented 4 years ago

I always got those warnings about libjpeg, but it never failed the build 😕

Yes, it is outdated, sorry, I will change it, is just in case your libjpeg folder is in other place.

My version is 1.39.4

nicolocarpignoli commented 4 years ago

the build does not failed actually. I see that the 'output' directory is not present. I have manually created it but does not work either

Carnaux commented 4 years ago

The dev branch just generates the NftMarkerCreator.min.js, to generate the marker itself you have to use the master

nicolocarpignoli commented 4 years ago

Yes, I was just to write it! I didn't understand that!

nicolocarpignoli commented 4 years ago

So the complete flux, as for now, should be:

I got error with this flow.

TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received type undefined

Also, I was able to create .iset files using the Editor.html from dev branch but they did not seem to work.

Carnaux commented 4 years ago

That flow is just if you want to make some changes to the min.js, the master branch is ready to use. I'm making some changes in the dev branch, making it cleaner, the html editor is discontinued.

Carnaux commented 4 years ago

I made the changes in the dev branch, it is cleaner now.

One thing I've been thinking is to merge the .iset, .fset and .fset3 files in just one maybe is more practical? What do you guys think? @kalwalt @ThorstenBux @nicolocarpignoli

jamess922 commented 4 years ago

Hi I tried with the pinball.jpg and it's come out with this C:\Users\Cai-TP>node app.js The EXIF info of this image is empty or it does not exist. Do you want to inform its properties manually?[y/n]

Carnaux commented 4 years ago

You type "y" and press enter after that will have other dialogs, for Width and Height of the image, and for DPI, if you want to inform one(or just press enter, the code will use 72 as default)

jamess922 commented 4 years ago

Thank you It's working ,get the files

nicolocarpignoli commented 4 years ago

it works on master branch! Great!!

As next step I would suggest to automate it even more:

For each of them, if can't retrieve, fallback to a default.

I don't know if this is all possible but should be great.

Anyway, your idea to create a basic HTML page as generator was great. This because people used to use a marker generator for fiducial marker on AR.js https://jeromeetienne.github.io/AR.js/three.js/examples/marker-training/examples/generator.html

Something very similar to create NFT images will do, like:

What do you think?

nicolocarpignoli commented 4 years ago

Next weeks I will be a bit busy to try to port NFT jsartoolkit5 on AR.js but I will always provide support for this, and when I will be more free even 'development' support :)

kalwalt commented 4 years ago

I think a node app is the most reasonable solution at the moment, much easier than having to clone artoolkitx, install all the dependencies (what a panic!) and compile the C++ project . Above all if you are not an experienced C++ developer this is discouraging...