CMD API deepcl_predict

[merceyz]

Back again, on this page https://github.com/hughperkins/DeepCL/blob/master/doc/Commandline.md It says to use "predict" to run a prediction but doesn't really say much more

Say I have 9 jpeg images somewhere that it has never trained on (not validation.txt) how would i proceed? IE. What arguments to give deepcl_predict?

[hughperkins]

something like:

./deepcl_predict inputfile=/tmp/t10k-images-idx3-ubyte writelabels=1 outputfile=predictions.txt

(but you'd use a manifest file in place of /tmp/t10k-images-idx3-ubyte)

[hughperkins]

Hi Chris, to what extent does this work for you? Any issues encountered?

[merceyz]

Hello,

I didn't get around to testing it until now but sadly it doesn't work

[hughperkins]

Ok. Will check. Hmmm, youre on Windows. That means my fix for relative directory wont work for you yet. I will check this point too...

[hughperkins]

predict should work now, in source, though I should probably create a new binary release for this.

[hughperkins]

(ie change in f4ec8f2 )

[merceyz]

Since i'm not quite sure how to build it myself i'll have to wait for the binary release

[merceyz]

I saw the issue was with labels in the changes you made so i added labels to the file, still same error

[hughperkins]

Hmmm. What if you put inputfile=test.txt ?

[merceyz]

Same error. It finds the file so that isn't the issue, if i specify one that doesn't exist it says it can't open it

deepcl_predict weightsfile=weights.dat inputfile=test.txt writelabels=1 outputfile=predictions.txt

[hughperkins]

Oh.... I bet it's because I'm reading 1024 bytes to look for the header, and test.txt is less than 1024 bytes. Do you want to confirm that if you add a few more lines to test.txt, so it's at least eg ~2KB, the error goes away? If so, I can look at fixing that.

[merceyz]

I added more images to the list which caused this output: However right after this it crashed instantly

[hughperkins]

Ok. I've addressed the file-length issue in 88d418b

[merceyz]

That +1 will start reading from "lanes=" instead of "planes=" if i'm not mistaken as you have a space in your sigString

So it will try to match

# format=deepcl-jpeg-list-v1 with # format=deepcl-jpeg-list-v1 p

Any clue about the crash?

[hughperkins]

That line is only to identify what file format it is, ie is it a manifest file? or is it a mnist datafile? and so on. The file is opened, a few bytes are read, then the file is closed again. To read the geometry, the file is re-opened later on:

    ifstream infile(imagesFilepath);
    char lineChars[1024];
    infile.getline(lineChars, 1024); // skip first, header, line
    string firstLine = string(lineChars);
//    cout << "firstline: [" << firstLine << "]" << endl;
    vector<string> splitLine = split(firstLine, " ");
    N = readIntValue(splitLine, "N");
    planes = readIntValue(splitLine, "planes");

[merceyz]

Yeah but since sigString has a space in it at the end it's length is 29, when it then adds one more to it it will end up comparing

# format=deepcl-jpeg-list-v1 to # format=deepcl-jpeg-list-v1 p

Which will be false and thus not match

[hughperkins]

ah, I see what you mean. Well... the next line sets that byte to a null string-terminator byte:

headerBytes[sigString.length()] = 0;

[merceyz]

Wont it still fail though because it's longer than the other? I obviously don't code c++ so i got no clue

Anyways, i went into the event viewer to get some details for it crashing but wasn't of much help

Faulting application name: deepcl_predict.exe, version: 0.0.0.0, time stamp: 0x56f3d870 Faulting module name: MSVCR100.dll, version: 10.0.30319.1, time stamp: 0x4ba220dc Exception code: 0xc0000005 Fault offset: 0x000000000003c04d Faulting process id: 0x23bc Faulting application start time: 0x01d1e5e8a3a7f3cf

[hughperkins]

Well... this particular bit should be identical on linux and windows, and the tests pass on linux, so I'm fairly sure this particular bit is ok. But, to answer your question about string length.... strings in C/C++ can take multiple forms. One form is null-terminated, which comprises one byte for each character in the string, and then one additional byte, which is set to 0, which is used to indicate the end of the string. The length of the string, as returned by eg strlen excludes that final null-byte. The length of the array containing that string is one more than the strlen length of the string.

As far as the crash after that, well, I'm not sure, I'm working through building on Windows, and then I'll try it.

[hughperkins]

I'm making a new build for you to try. Are you on 32-bit or 64-bit windows?

[hughperkins]

well... there are two versions built actually:

• http://deepcl.hughperkins.com/Downloads/deepcl-win64-v8.3.2alpha4.zip
• http://deepcl.hughperkins.com/Downloads/deepcl-win32-v8.3.2alpha4.zip

Do you want to try it, and see if anything improves and/or what the new error messages are?

[merceyz]

Sorry, I went to bed, was getting late here.

It worked now so great job! It almost predicted all of them correctly, 3/4, it did however output it as 0 and 1, how would i make it output the actual value (0.35, 0.67 etc)

For future references i'm on windows 7 64-bit.

I noticed that if I put in more images in the manifest than N specifies it crashes so perhaps a check against that.

[hughperkins]

It worked now so great job! It almost predicted all of them correctly, 3/4, it did however output it as 0 and 1, how would i make it output the actual value (0.35, 0.67 etc)

remove writelabels=1. Note that I didnt test this last night, so it might have some buggettes remaining. I might check this point...

For future references i'm on windows 7 64-bit.

Cool. Thanks!

I noticed that if I put in more images in the manifest than N specifies it crashes so perhaps a check against that

What do you mean by 'crash'? Do you mean, dies, with a message saying they dont match? Or pops up some kind of 'abort/debug' dialog? (The latter would be ugly, and undesirable. The former is intentional:

    if(n != N) {
        throw runtime_error("Error: number of images declared in manifest " + toString(N) + " doesnt match number actually in manifest " + toString(n));
    }

You could say I could detect the number automatically? I suppose that's true... do you prefer that the number is detect automatically, and you no longer need to put it in the first line of the manifest?

[merceyz]

remove writelabels=1. Note that I didnt test this last night, so it might have some buggettes remaining. I might check this point...

Cheers, worked fine. Detected 8/9 correctly, but having the value and not the label i can calculate my way to get the 9th correctly marked

What do you mean by 'crash'? Do you mean, dies, with a message saying they dont match? Or pops up some kind of 'abort/debug' dialog? (The latter would be ugly, and undesirable. The former is intentional:

Number of images in manifest is 10, N=9 it will crash as in screenshot above Remove the 10th image and it works or change N=10 and it works. If the number of images is 9 and N=10 it will throw the error and not crash

You could say I could detect the number automatically? I suppose that's true... do you prefer that the number is detect automatically, and you no longer need to put it in the first line of the manifest?

That would be useful yes and would just be to count the number of lines -1

[hughperkins]

Ah, right, thats pretty ugly :-D Ah, because it over-runs the end of the array....

That would be useful yes and would just be to count the number of lines - 1

Alright, let's run with that then. I will take a look.

[hughperkins]

(should be fixed in 254ddca ideally; building now; should pop out at http://deepcl.hughperkins.com/Downloads/deepcl-win64-v8.4.0alpha1.zip , once its done)

[hughperkins]

(oh, hmmm, the build failed, I will check...)

[merceyz]

I'll take a look at it tomorrow(later today) when i wake up, for now it's too late to be up

[merceyz]

I tested deepcl_train and removed the "TestFolder/" from all images in the manifest and it can't manage to locate the file even though datadir is set to TestFolder

So it seems that 452a6c9af883e8bb264ce9c0a69d0b63de42889f was unsucessful

[hughperkins]

Hmmm good point. Addressed in ddc6126 Can you try http://deepcl.hughperkins.com/Downloads/deepcl-win64-v8.4.0alpha4.zip ?

[hughperkins]

(v8.5.1 available http://deepcl.hughperkins.com/Downloads )

[merceyz]

Tried with 8.5.1 64-bit

All works fine, both N and "TestFolder/" could be removed

Thank you

[hughperkins]

Cool :-)