search

slackapi / java-slack-sdk

Slack Developer Kit (including Bolt for Java) for any JVM language
https://tools.slack.dev/java-slack-sdk/
MIT License
577 stars 214 forks source link

A recursive message reference causes a JSON data parsing error #1207

Closed jdlee726 closed 1 year ago

jdlee726 commented 1 year ago

(Describe your issue and goal here)

Reproducible in:

having the follwing json string for an message from slack platform. apparently, there is no 'type' field in response, but, GsonLayoutBlockFactory.deserialize() requires "type" field.

{
  "team": "<Team ID>",
  "channel": "<Channel ID>",
  "ts": "<timestamp>",
  "message": {
    "blocks": [
      {
        "type": "rich_text",
        "block_id": "rVD",
        "elements": [
          {
            "type": "rich_text_section",
            "elements": [
              {
                "type": "text",
                "text": "TEXT\n"
              }
            ]
          },
          {
            "type": "rich_text_list",
            "elements": [
              {
                "type": "rich_text_section",
                "elements": [
                  {
                    "type": "text",
                    "text": "TEXT"
                  }
                ]
              },
              {
                "type": "rich_text_section",
                "elements": [
                  {
                    "type": "text",
                    "text": "TEXT"
                  }
                ]
              },
              {
                "type": "rich_text_section",
                "elements": [
                  {
                    "type": "text",
                    "text": "TEXT"
                  }
                ]
              },
              {
                "type": "rich_text_section",
                "elements": [
                  {
                    "type": "text",
                    "text": "TEXT"
                  }
                ]
              }
            ],
            "style": "ordered",
            "indent": 0,
            "border": 0
          }
        ]
      }
    ]
  }
}

The Slack SDK version

google gson 2.10.1 

[INFO] +- com.slack.api:slack-api-client:jar:1.32.0:compile
[INFO] |  +- com.slack.api:slack-api-model:jar:1.32.0:compile
[INFO] +- com.slack.api:bolt:jar:1.32.0:compile
[INFO] |  +- com.slack.api:slack-app-backend:jar:1.32.0:compile
[INFO] +- com.slack.api:bolt-socket-mode:jar:1.32.0:compile
[INFO] +- com.slack.api:bolt-jakarta-servlet:jar:1.32.0:compile
[INFO] +- com.slack.api:slack-api-client:jar:1.32.0:compile
[INFO] |  +- com.slack.api:slack-api-model:jar:1.32.0:compile
[INFO] +- com.slack.api:bolt:jar:1.32.0:compile
[INFO] |  +- com.slack.api:slack-app-backend:jar:1.32.0:compile
[INFO] +- com.slack.api:bolt-jakarta-servlet:jar:1.32.0:compile
[INFO] |  +- com.slack.api:slack-api-client:jar:1.32.0:compile
[INFO] |  |  +- com.slack.api:slack-api-model:jar:1.32.0:compile
[INFO] |  +- com.slack.api:bolt:jar:1.32.0:compile
[INFO] |  |  \- com.slack.api:slack-app-backend:jar:1.32.0:compile
[INFO] |  \- com.slack.api:bolt-jakarta-servlet:jar:1.32.0:compile
[INFO] |  |  +- com.slack.api:slack-api-client:jar:1.32.0:compile
[INFO] |  |  |  +- com.slack.api:slack-api-model:jar:1.32.0:compile
[INFO] |  |  +- com.slack.api:bolt:jar:1.32.0:compile
[INFO] |  |  |  \- com.slack.api:slack-app-backend:jar:1.32.0:compile
[INFO] |  |  \- com.slack.api:bolt-jakarta-servlet:jar:1.32.0:compile

Java Runtime version

openjdk 20.0.2 2023-07-18 OpenJDK Runtime Environment (build 20.0.2+9-78) OpenJDK 64-Bit Server VM (build 20.0.2+9-78, mixed mode, sharing)

OS info

ProductName: macOS ProductVersion: 13.5.2 BuildVersion: 22G91 Darwin Kernel Version 22.6.0: Wed Jul 5 22:21:56 PDT 2023; root:xnu-8796.141.3~6/RELEASE_X86_64

Steps to reproduce:

methodClient.conversationsHistory()

Expected result:

normal conversion history

Actual result:

call stack

java.lang.NullPointerException: Cannot invoke "com.google.gson.JsonPrimitive.getAsString()" because "prim" is null
    at com.slack.api.util.json.GsonLayoutBlockFactory.deserialize(GsonLayoutBlockFactory.java:29)
    at com.slack.api.util.json.GsonLayoutBlockFactory.deserialize(GsonLayoutBlockFactory.java:12)
    at com.google.gson.internal.bind.TreeTypeAdapter.read(TreeTypeAdapter.java:76)
    at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:40)
    at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:82)
    at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:61)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.readIntoField(ReflectiveTypeAdapterFactory.java:212)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$FieldReflectionAdapter.readField(ReflectiveTypeAdapterFactory.java:433)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:393)
    at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:40)
    at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:82)
    at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:61)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.readIntoField(ReflectiveTypeAdapterFactory.java:212)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$FieldReflectionAdapter.readField(ReflectiveTypeAdapterFactory.java:433)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:393)
    at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:40)
    at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:82)
    at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:61)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.readIntoField(ReflectiveTypeAdapterFactory.java:212)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$FieldReflectionAdapter.readField(ReflectiveTypeAdapterFactory.java:433)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:393)
    at com.google.gson.Gson.fromJson(Gson.java:1227)
    at com.google.gson.Gson.fromJson(Gson.java:1137)
    at com.google.gson.Gson.fromJson(Gson.java:1047)
    at com.google.gson.Gson.fromJson(Gson.java:982)
    at com.slack.api.methods.impl.MethodsClientImpl.parseJsonResponseAndRunListeners(MethodsClientImpl.java:3539)
    at com.slack.api.methods.impl.MethodsClientImpl.parseJsonResponseAndRunListeners(MethodsClientImpl.java:3526)
    at com.slack.api.methods.impl.MethodsClientImpl.postFormWithTokenAndParseResponse(MethodsClientImpl.java:3433)
    at com.slack.api.methods.impl.MethodsClientImpl.conversationsHistory(MethodsClientImpl.java:1796)
    at com.slack.api.methods.impl.MethodsClientImpl.conversationsHistory(MethodsClientImpl.java:1801)

Requirements

seratch commented 1 year ago

Hi @jdlee726, thanks for reporting this issue. I tried reproducing the situation using the JSON data you provided, but parsing the data did not fail for me. Can you double-check if there might be a different data pattern causing the issue?

jdlee726 commented 1 year ago

when I debug at the follwing break-point, variable 'prim' is null, then cause NPE.

image
seratch commented 1 year ago

Yeah, I understand this could happen for sure but wanted to know the input causing the issue.

jdlee726 commented 1 year ago

FYI, the value of parameter 'json'

image
seratch commented 1 year ago

Thanks. This never happens to me, and we have never received this feedback in the last few years since this project started. Would it be possible to share more information to understand the cause of your issue, such as the code (how to initialize the API client, etc.)? It seems that Json parsers might not be working properly with your settings but I am unsure what the cause is.

jdlee726 commented 1 year ago

thanks and sure, but security reasons, I can share some code snipeets as the following pseudo codes. sorry for this. (more elaborated reproducible test case. using junit, vavr and lombok.)

    @Test
    public void bug_NPE() {
        Try.run(() -> {
            var config = new SlackConfig();
            config.setPrettyResponseLoggingEnabled(true);
            var slack = Slack.getInstance(config);
            var token = "<TOKEN-VALUE>";
            var methodsClient = slack.methods(token);

            var channelId = "<ChannelId>";
            var list = Try.of(() -> methodsClient.conversationsHistory(b -> b.channel(channelId)).getMessages()).get();
            log.info("list: {}", list.size());
        }).get();
    }
seratch commented 1 year ago

Thanks for sharing the details. I tried to reproduce the error with JDK 20 + Vavr but still no luck. The NPE itself is just an outcome of the GSON misbehavior (please note that I am still unsure if this could be an issue on GSON side), so avoiding the exception is not a solution for you. Even if I change the code to not throw the exception, your code still fails to load the API response.

I'm sorry to say this, but I don't have any further ideas to figure out the cause of your issue now. Please let us know whenever you find anything new on this.

jdlee726 commented 1 year ago

I tried with java 8 ( eclipse temurin-1.8.0_382 ), and getting almost the same call stack.

java.lang.NullPointerException
    at com.slack.api.util.json.GsonLayoutBlockFactory.deserialize(GsonLayoutBlockFactory.java:29)
    at com.slack.api.util.json.GsonLayoutBlockFactory.deserialize(GsonLayoutBlockFactory.java:12)
    at com.google.gson.internal.bind.TreeTypeAdapter.read(TreeTypeAdapter.java:76)
    at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:40)
    at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:82)
    at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:61)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.readIntoField(ReflectiveTypeAdapterFactory.java:212)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$FieldReflectionAdapter.readField(ReflectiveTypeAdapterFactory.java:433)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:393)
    at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:40)
    at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:82)
    at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:61)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.readIntoField(ReflectiveTypeAdapterFactory.java:212)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$FieldReflectionAdapter.readField(ReflectiveTypeAdapterFactory.java:433)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:393)
    at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:40)
    at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:82)
    at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:61)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.readIntoField(ReflectiveTypeAdapterFactory.java:212)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$FieldReflectionAdapter.readField(ReflectiveTypeAdapterFactory.java:433)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:393)
    at com.google.gson.Gson.fromJson(Gson.java:1227)
    at com.google.gson.Gson.fromJson(Gson.java:1137)
    at com.google.gson.Gson.fromJson(Gson.java:1047)
    at com.google.gson.Gson.fromJson(Gson.java:982)
    at com.slack.api.methods.impl.MethodsClientImpl.parseJsonResponseAndRunListeners(MethodsClientImpl.java:3539)
    at com.slack.api.methods.impl.MethodsClientImpl.parseJsonResponseAndRunListeners(MethodsClientImpl.java:3526)
    at com.slack.api.methods.impl.MethodsClientImpl.postFormWithTokenAndParseResponse(MethodsClientImpl.java:3433)
    at com.slack.api.methods.impl.MethodsClientImpl.conversationsHistory(MethodsClientImpl.java:1796)
    at com.slack.api.methods.impl.MethodsClientImpl.conversationsHistory(MethodsClientImpl.java:1801)
jdlee726 commented 1 year ago

I finally success to reproduce the error with the following public slack workspace.

https://app.slack.com/client/T05SQCT1N7Q/C05SQCT2RPG?ssb_vid=c989ce123d730cd76eed9470df574d7a

package bug.slack.npe;

import com.slack.api.Slack;
import com.slack.api.SlackConfig;
import com.slack.api.methods.MethodsClient;
import com.slack.api.model.Message;
import io.vavr.control.Try;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.exception.ExceptionUtils;
import org.junit.jupiter.api.Test;

import java.util.List;

@Slf4j
public class BugNpeTest {
    @Test
    public void bug_npe() {
        SlackConfig config = new SlackConfig();
        config.setPrettyResponseLoggingEnabled(true);
        Slack slack = Slack.getInstance(config);
        String token = "xoxp-***";
        MethodsClient methodsClient = slack.methods(token);

        String channelId = "C05SQCT2RPG";
        List<Message> list = Try.of(() -> methodsClient.conversationsHistory(b -> b.channel(channelId)).getMessages())
                .onFailure(e->log.error(ExceptionUtils.getStackTrace(e)))
                .get();
        log.info("list: {}", list.size());
    }
}
jdlee726 commented 1 year ago

with pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>slack_bug_npe</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>com.slack.api</groupId>
            <artifactId>slack-api-client</artifactId>
            <version>1.32.0</version>
        </dependency>

        <dependency>
            <groupId>io.vavr</groupId>
            <artifactId>vavr</artifactId>
            <version>0.10.4</version>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>1.18.28</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>1.7.36</version>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>3.12.0</version>
        </dependency>

        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>5.10.0</version>
            <scope>test</scope>
        </dependency>

    </dependencies>
</project>
seratch commented 1 year ago

@jdlee726 Thanks for sharing this. However, we are unable to use your API tokens to reproduce the error. Perhaps the token should have already been revoked by our crawler, which detects token exposure on the internet. If possible, could you enable debug-level logging and view the response data (i.e., JSON data)? Sharing the JSON with me can be helpful in identifying the possible cause of the error.

jdlee726 commented 1 year ago

slack automatically disabled the token because of publically available.

jdlee726 commented 1 year ago

If you give me some email adress, I will invite to this workspace

jdlee726 commented 1 year ago

json value already described at the first comment.

jdlee726 commented 1 year ago

the channel messages cause error.

image

seratch commented 1 year ago

@jdlee726 Thanks to your latest reply, I managed to reproduce the error. Here is an example conversations.history API response to reproduce the error:

{
  "ok": true,
  "oldest": "1695106171.334469",
  "messages": [
    {
      "client_msg_id": "xxx",
      "type": "message",
      "text": "test\n<https://xxx.slack.com/archives/C123/p1695106171334469>",
      "user": "U123",
      "ts": "1695106171.334469",
      "blocks": [
        {
          "type": "rich_text",
          "block_id": "Qkc",
          "elements": [
            {
              "type": "rich_text_section",
              "elements": [
                {
                  "type": "text",
                  "text": "test\n"
                },
                {
                  "type": "link",
                  "url": "https://xxx.slack.com/archives/C123/p1695106171334469"
                }
              ]
            }
          ]
        }
      ],
      "team": "T123",
      "edited": {
        "user": "U123",
        "ts": "1695106197.000000"
      },
      "attachments": [
        {
          "from_url": "https://xxx.slack.com/archives/C123/p1695106171334469",
          "ts": "1695106171.334469",
          "author_id": "U123",
          "channel_id": "C123",
          "channel_team": "T123",
          "is_msg_unfurl": true,
          "message_blocks": [
            {
              "team": "T123",
              "channel": "C123",
              "ts": "1695106171.334469",
              "message": {
                "blocks": [
                  {
                    "type": "rich_text",
                    "block_id": "Qkc",
                    "elements": [
                      {
                        "type": "rich_text_section",
                        "elements": [
                          {
                            "type": "text",
                            "text": "test\n"
                          },
                          {
                            "type": "link",
                            "url": "https://xxx.slack.com/archives/C123/p1695106171334469"
                          }
                        ]
                      }
                    ]
                  }
                ]
              }
            }
          ],
          "id": 1,
          "original_url": "https://xxx.slack.com/archives/C123/p1695106171334469",
          "fallback": "fallback",
          "text": "test\n<https://xxx.slack.com/archives/C123/p1695106171334469>",
          "author_name": "Kaz",
          "author_link": "https://xxx.slack.com/team/U123",
          "author_icon": "https://avatars.slack-edge.com/2023-09-19/5927659638721_20cd515c6ad81ae7b17a_48.jpg",
          "author_subname": "Kaz Sera",
          "mrkdwn_in": [
            "text"
          ],
          "footer": "Slack conversation"
        }
      ]
    }
  ],
  "has_more": false,
  "pin_count": 0,
  "channel_actions_ts": 1638492970,
  "channel_actions_count": 0
}

We will fix this issue as soon as possible and then release a patch. Thank you again for taking the time to report this and we apologize for the disruption.

seratch commented 1 year ago

If you need an immediate workaround, please downgrade this library to version 1.30.0. The bug was introduced in v1.31.0.

jdlee726 commented 1 year ago

Very Helpful! With your support, I can leave work on time today!