Nil Pointer Dereference During Lookup Join

[wduczkowski]

Environment:

• eKuiper version: 1.14.6-full
• Hardware configuration (e.g. lscpu):
• OS (e.g. cat /etc/os-release): Debian GNU/Linux 11 (bullseye)"
• Others: deploymen as docker container, image version: docker pull lfedge/ekuiper:v1.14.6-full

We are encountering a runtime error when attempting to enrich sensor data with lookup table data in the enrichSesnorWithLookupRoom rule. The error indicates a nil pointer dereference in the lookup node of the pipeline.

Steps to Reproduce:

1. Setup Aggregated Table: Aggregated sensor data from the ekuiperSensor stream using the SQL query:

   
   SELECT 
     key,
     LATEST(temperature) AS temperature,
     LATEST(humidity) AS humidity,
     LATEST(pressure) AS pressure
   FROM 
     ekuiperSensor
   GROUP BY key, TUMBLINGWINDOW(ss, 60);

  Data is sent to the ekuiper.sensor.aggregation topic.

2.  **Setup Lookup Table:** A lookup table (lookupTable) is configured to pull data from an HTTP REST endpoint:
table config:

{ "Name": "lookupTable", "StreamFields": null, "Options": { "key": "deviceId", "format": "json", "confKey": "lookupTable", "type": "httppull", "kind": "lookup" }, "StreamType": 1, "Statement": null }

source config key:

{ "lookupTable": { "bodyType": "json", "incremental": true, "insecureSkipVerify": false, "interval": 10000, "method": "get", "oauth": { "access": {}, "refresh": {} }, "responseType": "code", "timeout": 500, "url": "http://192.168.162.94:3000/room" } }

response from REST API

[ { "roomId": "ae440646-95a3-4367-b3a7-7fc836111d68", "deviceId": "zigbee2mqtt/TermometrSalon", "id": "zigbee2mqtt/TermometrSalon" }, { "roomId": "542a7f00-f1d7-4ff1-a6c5-022187638daa", "deviceId": "DEV-1", "id": "DEV-1" }, { "roomId": "41c8df49-44bc-4aaf-a3d5-66e1fda95919", "deviceId": "DEV-2", "id": "DEV-2" }, { "roomId": "e4f94c67-b37c-495d-a4e4-2801bdb84303", "deviceId": "DEV-3", "id": "DEV-3" }, { "roomId": "04d68026-2287-49c9-bec3-bb522af37088", "deviceId": "DEV-4", "id": "DEV-4" }, { "roomId": "80d96222-cdc3-4dae-a89f-d95c815a6288", "deviceId": "DEV-6", "id": "DEV-6" }, { "roomId": "145171df-aee4-4f2c-b453-ff3fc4e200cb", "deviceId": "DEV-7", "id": "DEV-7" }, { "roomId": "8862519d-72b2-42e2-967b-aa71b64b4b09", "deviceId": "DEV-8", "id": "DEV-8" }, { "roomId": "2975daea-3b94-4c43-bedf-e21f4d7326a3", "deviceId": "DEV-9", "id": "DEV-9" }, { "roomId": "08d8c0b4-c286-4fdf-9239-2b2667d500bb", "deviceId": "DEV-10", "id": "DEV-10" }, { "roomId": "ae440646-95a3-4367-b3a7-7fc836111d68", "deviceId": "DEV-11", "id": "DEV-11" }, { "roomId": "08d8c0b4-c286-4fdf-9239-2b2667d500bb", "deviceId": "TempHumPres", "id": "TempHumPres" }, { "roomId": "e4f94c67-b37c-495d-a4e4-2801bdb84303", "deviceId": "iPhone1", "id": "iPhone1" }, { "roomId": "e4f94c67-b37c-495d-a4e4-2801bdb84303", "deviceId": "iPhone2", "id": "iPhone2" } ]

3. **Sensor Stream**: Sample data in the ekuiperSensor stream:

{"humidity": 48.2, "key": "TempHumPres", "topic": "TempHumPres/get/sensor/2"} {"key": "TempHumPres", "temperature": 22.4, "topic": "TempHumPres/get/sensor/1"} {"key": "TempHumPres", "pressure": 1005.7, "topic": "TempHumPres/get/sensor/3"}

4. **Enrichment Rule**: The following SQL rule is used to join the aggregated sensor data with the lookup table:

SELECT ekuiperSensorAggregation.humidity, ekuiperSensorAggregation.pressure, ekuiperSensorAggregation.temperature, ekuiperSensorAggregation.key, lookupTable.deviceId, lookupTable.roomId FROM ekuiperSensorAggregation LEFT JOIN lookupTable ON ekuiperSensorAggregation.key = lookupTable.deviceId;

5. **Runtime Error**: The following error appears in the logs:

time="2024-11-20T10:24:00Z" level=error msg="runtime error: runtime error: invalid memory address or nil pointer dereference" file="infra/saferun.go:56" rule=enrichSesnorWithLookupRoom goroutine 49152 [running]: runtime/debug.Stack() runtime/debug/stack.go:24 +0x5e runtime/debug.PrintStack() runtime/debug/stack.go:16 +0x13 time="2024-11-20T10:24:00Z" level=error msg="closing rule enrichSesnorWithLookupRoom for error: runtime error: invalid memory address or nil pointer dereference" file="rule/ruleState.go:242" github.com/lf-edge/ekuiper/pkg/infra.SafeRun.func1() github.com/lf-edge/ekuiper/pkg/infra/saferun.go:34 +0x3d panic({0x2a70180?, 0x5310480?}) runtime/panic.go:770 +0x132 github.com/lf-edge/ekuiper/internal/io/http.(ClientConf).parseResponse(0xc001a03c20, {0x34f87f0, 0xc00045ef20}, 0xc0012c4630, 0x70?, 0x0, 0x0) github.com/lf-edge/ekuiper/internal/io/http/client.go:417 +0x345 github.com/lf-edge/ekuiper/internal/io/http.(lookupSource).pull(0xc000f1c138, {0x34f87f0, 0xc00045ef20}) github.com/lf-edge/ekuiper/internal/io/http/httppull_lookup.go:109 +0x75c github.com/lf-edge/ekuiper/internal/io/http.(lookupSource).Lookup(0x1?, {0x34f87f0?, 0xc00045ef20?}, {0x60?, 0x50?, 0xc000bb0c60?}, {0xc000b6b010, 0x1, 0xc00179ff20?}, {0xc0008a9f40, ...}) github.com/lf-edge/ekuiper/internal/io/http/httppull_lookup.go:47 +0x3f github.com/lf-edge/ekuiper/internal/topo/node.(LookupNode).lookup(0xc000a2ce80, {0x34f87f0, 0xc00045ef20}, {0x34e48e8, 0xc0007ba400}, 0xc00011c4a8, {0x34d48c8, 0xc000f1c138}, 0xc00179ff20, 0x0) github.com/lf-edge/ekuiper/internal/topo/node/lookup_node.go:201 +0x423 github.com/lf-edge/ekuiper/internal/topo/node.(LookupNode).Exec.func1.1() github.com/lf-edge/ekuiper/internal/topo/node/lookup_node.go:121 +0xa5e github.com/lf-edge/ekuiper/pkg/infra.SafeRun(0xc000a07500?) github.com/lf-edge/ekuiper/pkg/infra/saferun.go:45 +0x57 github.com/lf-edge/ekuiper/internal/topo/node.(LookupNode).Exec.func1() github.com/lf-edge/ekuiper/internal/topo/node/lookup_node.go:83 +0x7a created by github.com/lf-edge/ekuiper/internal/topo/node.(*LookupNode).Exec in goroutine 49150 github.com/lf-edge/ekuiper/internal/topo/node/lookup_node.go:82 +0x1bd time="2024-11-20T10:24:00Z" level=info msg="source ekuiperSensorAggregation done" file="node/source_node.go:142" rule=enrichSesnorWithLookupRoom

**Expected Behavior:**
The rule should successfully enrich the sensor data with the room and device information from the lookup table.

[ngjaying]

@wduczkowski What's the config of the httppull lookup table? Try to set incremental=false there.

[wduczkowski]

I have updated the issue description, added the table configuration, and httpull. I also tested the functionality with incremental=false, but the same error occurred.

[ngjaying]

I have updated the issue description, added the table configuration, and httpull. I also tested the functionality with incremental=false, but the same error occurred.

In the source confkey lookuptable, set "incremental": false. Looks like it is still true.

[wduczkowski]

After switching to incremental=false and restarting the rule , it started working. This is a bit strange because, in version ekuiper 2.0.0, the same case works with the incremental=true flag.

[ngjaying]

It is a defect and fixed in v2.