PoseNet Bug using custom trained model in a reactjs environment

Hey there! I'm trying to use ml5js in a react poseNet project i'm doing and even though i use the same inputs as in a similar yoga project i get this error when doing: let inputs = [] for (let i = 0; i < pose.keypoints.length; i++) { let x = pose.keypoints[i].position.x let y = pose.keypoints[i].position.y inputs.push(x) inputs.push(y) } brainRef.current.classify(inputs, gotResults) it somehow doesn't recognize the right inputs in the latest ml5js library and the results on gotResults are undefined.

error: TypeError: Cannot convert undefined or null to object at Function.keys (<anonymous>) at t.<anonymous> (index.js:1044:1) at l (runtime.js:63:1) at Generator._invoke (runtime.js:294:1) at http://Generator.next (runtime.js:119:1) at n (asyncToGenerator.js:3:1) at s (asyncToGenerator.js:25:1) at asyncToGenerator.js:32:1 at new Promise (<anonymous>) at t.<anonymous> (asyncToGenerator.js:21:1)

it says the error is in classifyInternal on the npm package (i'm using latest one). would you please be able to help?

Hey Miaoye,

You can see my code at: https://github.com/yehonatanyosefi/eramorph/blob/main/src/cmps/Pose.jsx

Feel free to email me for any more questions.

On Tue, Jun 27, 2023, 17:06 Miaoye Que @.***> wrote:

Hi Yehonatan,

Thank you for opening an issue with us! I'm wondering if you can point me to where the code lives or upload a more complete snippet (I'm assuming the code you shared is from asyncToGenerator.js?) so that I can try to reproduce the error?

Miaoye

— Reply to this email directly, view it on GitHub https://github.com/ml5js/ml5-library/issues/1468#issuecomment-1609697896, or unsubscribe https://github.com/notifications/unsubscribe-auth/A5N5RIBUGAH3KQEHBFO7HADXNLZI3ANCNFSM6AAAAAAZMNBREU . You are receiving this because you authored the thread.Message ID: @.***>

React stack traces are the worst. All I can tell is that somewhere inside classifyInternal (or any of the functions that it calls) there is an Object.keys(something) on a something which is not an object.

The only top-level Object.keys are on meta.inputs and meta.outputs. I don't know how those could be undefined but humor me and add some extra logging. The line we're looking for might be something else really deep in the tree and hard to find.

const classifyPose = async () => {
    try {
        if (pose && skeleton.length) {
            let inputs = []
            for (let i = 0; i < pose.keypoints.length; i++) {
                let x = pose.keypoints[i].position.x
                let y = pose.keypoints[i].position.y
                inputs.push(x)
                inputs.push(y)
            }
            console.log('Inputs', inputs);
            const meta = brainRef.current.neuralNetworkData.meta;
            console.log('Meta', meta);
            console.log('meta.inputs', meta.inputs);
            console.log('meta.outputs', meta.outputs);
            const results = await brainRef.current.classify(inputs);
            gotResults(undefined, results);
        } else {
            // console.log('Pose not found')
            setCurrPose('Not found')
            posesArray.current = [...posesArray.current, null]
        }
        handleGameTik()
    } catch(e) {
        console.log('Caught error', e);
        console.trace();
    }
}

Sorry for the late reply, I was on a vacation in Rome. Here are the results of the modified classify function: Inputs (34) [329.2766621511734, 232.7027044036509, 338.82978149889044, 221.55281749680813, 315.95720491520626, 221.3593028027724, 354.0813045279061, 235.36775878431268, 294.83786972580253, 236.0488698640222, 380.0362506079767, 315.43611311263146, 258.1640078770975, 306.9329822109831, 423.33587527738933, 422.2171400493221, 212.15757733653027, 393.2147017898263, 419.57896429277116, 500.38656865576365, 221.31209674048517, 470.4224481842398, 355.87284577959707, 505.9700231886096, 260.7915808532952, 494.00144940684277, 361.01693773084116, 581.7396539472884, 249.60313255220998, 576.3485622777085, 362.25384574919826, 571.0491598997598, 245.90564950430903, 566.701991456028] Meta:

data:
1. inputMax: Array(34)
  1. 0: 357.1703796386719
  2. 1: 359.7862243652344
  3. 2: 373.8389587402344
  4. 3: 358.5173034667969
  5. 4: 356.5716552734375
  6. 5: 356.08941650390625
  7. 6: 381.3825378417969
  8. 7: 357.2954406738281
  9. 8: 368.54266357421875
  10. 9: 359.2048034667969
  11. 10: 381.4326477050781
  12. 11: 377.7006530761719
  13. 12: 391.3138732910156
  14. 13: 379.7683410644531
  15. 14: 416.74847412109375
  16. 15: 390.48486328125
  17. 16: 420.24658203125
  18. 17: 389.9808044433594
  19. 18: 435.03515625
  20. 19: 428.47528076171875
  21. 20: 383.2144470214844
  22. 21: 435.2953796386719
  23. 22: 394.73492431640625
  24. 23: 419.23663330078125
  25. 24: 408.4559020996094
  26. 25: 420.2417297363281
  27. 26: 382.253662109375
  28. 27: 456.9807434082031
  29. 28: 369.4338073730469
  30. 29: 474.966796875
  31. 30: 437.54681396484375
  32. 31: 494.4590148925781
  33. 32: 417.29620361328125
  34. 33: 517.787841796875
  35. length: 34
2. inputMin: Array(34)
  1. 0: 214.403564453125
  2. 1: 96.04393005371094
  3. 2: 222.59112548828125
  4. 3: 90.27008819580078
  5. 4: 218.8485870361328
  6. 5: 90.7065200805664
  7. 6: 223.3074951171875
  8. 7: 97.40863800048828
  9. 8: 213.8081817626953
  10. 9: 96.97725677490234
  11. 10: 220.5511932373047
  12. 11: 137.7554931640625
  13. 12: 205.00909423828125
  14. 13: 139.93252563476562
  15. 14: 193.44581604003906
  16. 15: 118.66921997070312
  17. 16: 184.82797241210938
  18. 17: 101.13426971435547
  19. 18: 170.43374633789062
  20. 19: 53.16675567626953
  21. 20: 135.8551025390625
  22. 21: 51.93642044067383
  23. 22: 217.1484832763672
  24. 23: 231.117919921875
  25. 24: 212.82020568847656
  26. 25: 226.18161010742188
  27. 26: 195.08309936523438
  28. 27: 258.8846740722656
  29. 28: 176.20126342773438
  30. 29: 265.0129699707031
  31. 30: 164.59524536132812
  32. 31: 294.2119445800781
  33. 32: 166.31153869628906
  34. 33: 295.1418762207031
  35. length: 34
3. outputMax: 1
4. outputMin: 0
meta:
1. inputUnits: 34
2. inputs:
  1. input0: {dtype: 'number'}
  2. input1: {dtype: 'number'}
  3. input2: {dtype: 'number'}
  4. input3: {dtype: 'number'}
  5. input4: {dtype: 'number'}
  6. input5: {dtype: 'number'}
  7. input6: {dtype: 'number'}
  8. input7: {dtype: 'number'}
  9. input8: {dtype: 'number'}
  10. input9: {dtype: 'number'}
  11. input10: {dtype: 'number'}
  12. input11: {dtype: 'number'}
  13. input12: {dtype: 'number'}
  14. input13: {dtype: 'number'}
  15. input14: {dtype: 'number'}
  16. input15: {dtype: 'number'}
  17. input16: {dtype: 'number'}
  18. input17: {dtype: 'number'}
  19. input18: {dtype: 'number'}
  20. input19: {dtype: 'number'}
  21. input20: {dtype: 'number'}
  22. input21: {dtype: 'number'}
  23. input22: {dtype: 'number'}
  24. input23: {dtype: 'number'}
  25. input24: {dtype: 'number'}
  26. input25: {dtype: 'number'}
  27. input26: {dtype: 'number'}
  28. input27: {dtype: 'number'}
  29. input28: {dtype: 'number'}
  30. input29: {dtype: 'number'}
  31. input30: {dtype: 'number'}
  32. input31: {dtype: 'number'}
  33. input32: {dtype: 'number'}
  34. input33: {dtype: 'number'}
3. isNormalized: true
4. outputUnits: 6
5. outputs:
  1. output0:
    1. dtype: "string"
    2. legend:
      1. 1: (6) [1, 0, 0, 0, 0, 0]
      2. 2: (6) [0, 1, 0, 0, 0, 0]
      3. 3: (6) [0, 0, 1, 0, 0, 0]
      4. 4: (6) [0, 0, 0, 1, 0, 0]
      5. 5: (6) [0, 0, 0, 0, 1, 0]
      6. 6: (6) [0, 0, 0, 0, 0, 1]
    3. uniqueValues: Array(6)
      1. 0: "1"
      2. 1: "2"
      3. 2: "3"
      4. 3: "4"
      5. 4: "5"
      6. 5: "6"
      7. length: 6

meta.inputs undefined Pose.jsx:177 meta.outputs undefined Caught error TypeError: Cannot convert undefined or null to object at Function.keys () at t. (index.js:1044:1) at l (runtime.js:63:1) at Generator._invoke (runtime.js:294:1) at Generator.next (runtime.js:119:1) at n (asyncToGenerator.js:3:1) at s (asyncToGenerator.js:25:1) at asyncToGenerator.js:32:1 at new Promise () at t. (asyncToGenerator.js:21:1) Console Trace: overrideMethod @ react_devtools_backend_compact.js:2367 classifyPose @ Pose.jsx:188 await in classifyPose (async) handleStart @ Pose.jsx:73 callCallback @ react-dom.development.js:4164 invokeGuardedCallbackDev @ react-dom.development.js:4213 invokeGuardedCallback @ react-dom.development.js:4277 invokeGuardedCallbackAndCatchFirstError @ react-dom.development.js:4291 executeDispatch @ react-dom.development.js:9041 processDispatchQueueItemsInOrder @ react-dom.development.js:9073 processDispatchQueue @ react-dom.development.js:9086 dispatchEventsForPlugins @ react-dom.development.js:9097 (anonymous) @ react-dom.development.js:9288 batchedUpdates$1 @ react-dom.development.js:26140 batchedUpdates @ react-dom.development.js:3991 dispatchEventForPluginEventSystem @ react-dom.development.js:9287 dispatchEventWithEnableCapturePhaseSelectiveHydrationWithoutDiscreteEventReplay @ react-dom.development.js:6465 dispatchEvent @ react-dom.development.js:6457 dispatchDiscreteEvent On Wed, Jun 28, 2023 at 3:05 AM Linda Paiste @.***> wrote:

React stack traces are the worst. All I can tell is that somewhere inside classifyInternal https://github.com/ml5js/ml5-library/blob/f80f95aa6b31191ab7e79ff466e8506f5e48a172/src/NeuralNetwork/index.js#L1045 (or any of the functions that it calls) there is an Object.keys(something) on a something which is not an object.

The only top-level Object.keys are on meta.inputs and meta.outputs. I don't know how those could be undefined but humor me and add some extra logging. The line we're looking for might be something else really deep in the tree and hard to find.

const classifyPose = async () => { try { if (pose && skeleton.length) { let inputs = [] for (let i = 0; i < pose.keypoints.length; i++) { let x = pose.keypoints[i].position.x let y = pose.keypoints[i].position.y inputs.push(x) inputs.push(y) } console.log('Inputs', inputs); const meta = brainRef.current.neuralNetworkData.meta; console.log('Meta', meta); console.log('meta.inputs', meta.inputs); console.log('meta.outputs', meta.outputs); const results = await brainRef.current.classify(inputs); gotResults(undefined, results); } else { // console.log('Pose not found') setCurrPose('Not found') posesArray.current = [...posesArray.current, null] } handleGameTik() } catch(e) { console.log('Caught error', e); console.trace(); } }

— Reply to this email directly, view it on GitHub https://github.com/ml5js/ml5-library/issues/1468#issuecomment-1610385983, or unsubscribe https://github.com/notifications/unsubscribe-auth/A5N5RIBT6SM4IFVNRAS7UUTXNNYNTANCNFSM6AAAAAAZMNBREU . You are receiving this because you authored the thread.Message ID: @.***>

-- יהונתן יוספי

Okay we are getting somewhere! You’ve confirmed that meta.inputs and meta.outputs are both undefined and that’s what triggers the TypeError.

In looking at your meta object I can see that the info which we need is there but it’s not in the right place. It looks like it’s meta.meta.inputs instead of meta.inputs.

I haven’t got to the root problem yet. That is, why the meta is structured incorrectly. I’ll need to play around more and run your code. It’s either a problem with reading your metadata.json file or it’s a problem with exporting the model which led to an incorrect metadata.json. There probably exists a quick fix where I tell you what to change in the metadata.json file to make it work but that’s a poor solution that doesn’t address why it’s wrong. Did you use ml5 to generate the saved model that you are loading?

Yes, it's a ml5 model. I'd love help with it, i'll try playing with the nesting in the meta objects too, see if it works, but if you can do anything to help me figure it out faster, it'd be great! :)

On Sun, Jul 2, 2023 at 7:39 PM Linda Paiste @.***> wrote:

Okay we are getting somewhere! You’ve confirmed that meta.inputs and meta.outputs are both undefined and that’s what triggers the TypeError.

In looking at your meta object I can see that the info which we need is there but it’s not in the right place. It looks like it’s meta.meta.inputs instead of meta.inputs.

I haven’t got to the root problem yet. That is, why the meta is structured incorrectly. I’ll need to play around more and run your code. It’s either a problem with reading your metadata.json file or it’s a problem with exporting the model which led to an incorrect metadata.json. There probably exists a quick fix where I tell you what to change in the metadata.json file to make it work but that’s a poor solution that doesn’t address why it’s wrong. Did you use ml5 to generate the saved model that you are loading?

— Reply to this email directly, view it on GitHub https://github.com/ml5js/ml5-library/issues/1468#issuecomment-1616726247, or unsubscribe https://github.com/notifications/unsubscribe-auth/A5N5RIHOYE6BB6VKG7R63PDXOGP55ANCNFSM6AAAAAAZMNBREU . You are receiving this because you authored the thread.Message ID: @.***>

-- יהונתן יוספי

I took an old minified model someone used and it seems to work now. It seems new updates to ml5 library changed how it works and now legacy code is breaking when updating to it. I don't know at what version the minified library I found works, but it seems to be working fine now.

If there is a way for me to find the new syntaxes for using it in the posenet custom model, I'd love to use that.

On Sun, Jul 2, 2023 at 8:23 PM Yehonatan Yosefi @.***> wrote:

Yes, it's a ml5 model. I'd love help with it, i'll try playing with the nesting in the meta objects too, see if it works, but if you can do anything to help me figure it out faster, it'd be great! :)

On Sun, Jul 2, 2023 at 7:39 PM Linda Paiste @.***> wrote:

Okay we are getting somewhere! You’ve confirmed that meta.inputs and meta.outputs are both undefined and that’s what triggers the TypeError.

In looking at your meta object I can see that the info which we need is there but it’s not in the right place. It looks like it’s meta.meta.inputs instead of meta.inputs.

I haven’t got to the root problem yet. That is, why the meta is structured incorrectly. I’ll need to play around more and run your code. It’s either a problem with reading your metadata.json file or it’s a problem with exporting the model which led to an incorrect metadata.json. There probably exists a quick fix where I tell you what to change in the metadata.json file to make it work but that’s a poor solution that doesn’t address why it’s wrong. Did you use ml5 to generate the saved model that you are loading?

— Reply to this email directly, view it on GitHub https://github.com/ml5js/ml5-library/issues/1468#issuecomment-1616726247, or unsubscribe https://github.com/notifications/unsubscribe-auth/A5N5RIHOYE6BB6VKG7R63PDXOGP55ANCNFSM6AAAAAAZMNBREU . You are receiving this because you authored the thread.Message ID: @.***>

-- יהונתן יוספי

-- יהונתן יוספי

ml5js / ml5-library

PoseNet Bug using custom trained model in a reactjs environment #1468