search

apache / skywalking

APM, Application Performance Monitoring System
https://skywalking.apache.org/
Apache License 2.0
23.72k stars 6.5k forks source link

8.0.0-8.0.1。table:endpoint_relation_server_side,column:entity_id .Data too long for column 'entity_id' at row 1 #5013

Closed yazong closed 4 years ago

yazong commented 4 years ago

**Please answer these questions before submitting your issue.

Question

Bug

(一)测试环境测试机centos7-64安装skywalking8.0.1。jdk1.8-64bit。 CentOS Linux release 7.5.1804 (Core) 3.10.0-862.el7.x86_64

(二)本机PC-windows10-64bit基于jdk1.8-64bit使用IDEA2018.1-64bit版本启动Springcloud网关，配置如下:

Greenwich.SR3 org.springframework.boot spring-boot-starter-parent 2.1.9.RELEASE org.springframework.cloud spring-cloud-starter-gateway spring-cloud-starter-gateway(2.1.3.RELEASE) IDEA配置skywalking-agent.jar以及 SW_AGENT_NAME=service-gateway;SW_AGENT_COLLECTOR_BACKEND_SERVICES=10.10.8.17:11800 在网关本地启动完成，还未通过网关访问微服务时(比如http://172.50.3.249:8770/service-hi/hi)，即个人理解为初始加载这个工程时。 skywalking-oap-server.log日志中抛出如下异常: 2020-07-02 15:09:49,144 - org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO - 75 [pool-4-thread-1] ERROR [] - Data truncation: Data too long for column 'entity_id' at row 1 com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Data too long for column 'entity_id' at row 1 at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:97) ~[mysql-connector-java-6.0.6.jar:6.0.6] at com.mysql.cj.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:1299) ~[mysql-connector-java-6.0.6.jar:6.0.6] at com.mysql.cj.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:701) ~[mysql-connector-java-6.0.6.jar:6.0.6] at com.mysql.cj.jdbc.PreparedStatement.execute(PreparedStatement.java:1153) ~[mysql-connector-java-6.0.6.jar:6.0.6] at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44) ~[HikariCP-3.1.0.jar:?] at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java) ~[HikariCP-3.1.0.jar:?] at org.apache.skywalking.oap.server.storage.plugin.jdbc.SQLExecutor.invoke(SQLExecutor.java:53) ~[storage-jdbc-hikaricp-plugin-8.0.1.jar:8.0.1] at org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO.synchronous(H2BatchDAO.java:72) ~[storage-jdbc-hikaricp-plugin-8.0.1.jar:8.0.1] at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.extractDataAndSave(PersistenceTimer.java:122) ~[server-core-8.0.1.jar:8.0.1] at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$start$0(PersistenceTimer.java:78) ~[server-core-8.0.1.jar:8.0.1] at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33) [apm-util-8.0.1.jar:8.0.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_172] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_172] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_172] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172] 个人通过修改apache-skywalking-apm-bin/config/log4j2.xml的日志输出级别为DEBUG后，发现 在插入或更新endpoint_relation_server_side表时，发现接收的entity_id值过长，大于表中的varchar(200)。 1)我把其他表检查了一下，其他表的entity_id为varchar(512)，当我改成功512后就没再报错。 2)docs/en/setup/service-agent/java-agent/Supported-list.md支持 Spring Cloud Gateway 2.0.2.RELEASE -> 2.2.x.RELEASE (Optional²) 注: 本次本机测试为单节点eureka+单节点gateway+双节点service 不确定我提的这个issue是否正确，打扰了。谢谢。 ___ ### Requirement or improvement - Please describe about your requirements or improvement suggestions.**
wu-sheng commented 4 years ago

English only, please.

yazong commented 4 years ago

English only, please.

2020-07-02 15:09:49,144 - org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO - 75 [pool-4-thread-1] ERROR [] - Data truncation: Data too long for column 'entity_id' at row 1 com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Data too long for column 'entity_id' at row 1 at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:97) ~[mysql-connector-java-6.0.6.jar:6.0.6] at com.mysql.cj.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:1299) ~[mysql-connector-java-6.0.6.jar:6.0.6] at com.mysql.cj.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:701) ~[mysql-connector-java-6.0.6.jar:6.0.6] at com.mysql.cj.jdbc.PreparedStatement.execute(PreparedStatement.java:1153) ~[mysql-connector-java-6.0.6.jar:6.0.6] at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44) ~[HikariCP-3.1.0.jar:?] at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java) ~[HikariCP-3.1.0.jar:?] at org.apache.skywalking.oap.server.storage.plugin.jdbc.SQLExecutor.invoke(SQLExecutor.java:53) ~[storage-jdbc-hikaricp-plugin-8.0.1.jar:8.0.1] at org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO.synchronous(H2BatchDAO.java:72) ~[storage-jdbc-hikaricp-plugin-8.0.1.jar:8.0.1] at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.extractDataAndSave(PersistenceTimer.java:122) ~[server-core-8.0.1.jar:8.0.1] at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$start$0(PersistenceTimer.java:78) ~[server-core-8.0.1.jar:8.0.1] at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33) [apm-util-8.0.1.jar:8.0.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_172] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_172] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_172] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]

how do you solve this problem,please?

wu-sheng commented 4 years ago

I think this is code level bug.

  1. EndpointRelationServerSideMetrics#entityId
  2. ServiceInstanceRelationClientSideMetrics#entityId
  3. ServiceInstanceRelationServerSideMetrics#entityId
  4. ServiceRelationClientSideMetrics#entityId
  5. ServiceRelationServerSideMetrics#entityId

Should add length statement in the Column annotation, should be @Column(columnName = ENTITY_ID, length = 512)

Could you like to test locally? And send a pull request if the test passes?

zhousheng2 commented 4 years ago

I have the same problem @wu-sheng

2020-07-06 10:44:43,117 - org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO - 75 [pool-4-thread-1] ERROR [] - Data truncation: Data too long for column 'entity_id' at row 1 com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Data too long for column 'entity_id' at row 1 at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:104) ~[mysql-connector-java-8.0.20.jar:8.0.20] at com.mysql.cj.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:637) ~[mysql-connector-java-8.0.20.jar:8.0.20] at com.mysql.cj.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:418) ~[mysql-connector-java-8.0.20.jar:8.0.20] at com.mysql.cj.jdbc.ClientPreparedStatement.execute(ClientPreparedStatement.java:370) ~[mysql-connector-java-8.0.20.jar:8.0.20] at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44) ~[HikariCP-3.1.0.jar:?] at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java) ~[HikariCP-3.1.0.jar:?] at org.apache.skywalking.oap.server.storage.plugin.jdbc.SQLExecutor.invoke(SQLExecutor.java:53) ~[storage-jdbc-hikaricp-plugin-8.0.1.jar:8.0.1] at org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO.synchronous(H2BatchDAO.java:72) ~[storage-jdbc-hikaricp-plugin-8.0.1.jar:8.0.1] at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.extractDataAndSave(PersistenceTimer.java:122) ~[server-core-8.0.1.jar:8.0.1] at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$start$0(PersistenceTimer.java:78) ~[server-core-8.0.1.jar:8.0.1] at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33) [apm-util-8.0.1.jar:8.0.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_212] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_212] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_212] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]

zhousheng2 commented 4 years ago

I think it's a problem with the skywalking plugin

eg：2020-07-06 10:47:55,164 - org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO - 75 [pool-4-thread-1] ERROR [] - Data truncation: Data too long for column 'entity_id' at row 1

wu-sheng commented 4 years ago

@zhousheng2 I have provided the solution above, both of you are welcome to have a try and send the pull request.

zhousheng2 commented 4 years ago

@wu-sheng Thank you very much. At present, we also use version 8.0.1

wu-sheng commented 4 years ago

@zhousheng2 I am not using H2/MySQL/TiDB, so have to ask you to change source codes, and test locally.

yazong commented 4 years ago

thanks. my source code compile failed,i will continue to try and test.

wu-sheng commented 4 years ago

@yazong Why compiling fail? Have you read the document about compiling?

yazong commented 4 years ago

a small problem.i can solve.i'm doing other work.

yazong commented 4 years ago

@zhousheng2 this issue has been resolved. https://github.com/apache/skywalking/issues/5013

yazong commented 4 years ago

@wu-sheng

1) i found that the length value of these files was also set at 200. i did not see any related problems. so i was not sure,so i did not change it.

PercentileFunction HistogramFunction AvgHistogramFunction AvgFunction

2)

@ScopeDefaultColumn.VirtualColumnDefinition(fieldName = "entityId", columnName = "entity_id", isID = true, type = String.class) public class ServiceInstanceRelation extends Source {

@Target({ElementType.TYPE}) @Retention(RetentionPolicy.RUNTIME) public @interface VirtualColumnDefinition { String fieldName();

    String columnName();

    Class type();

    boolean isID() default false;

    /**
     * Define column length, only effective when the type is String.
     */
    int length() default 512;
}

why do the same fields have different lengths?

wu-sheng commented 4 years ago

Your formats are not well. It is very hard to read. For function(s), I think it still missed the length for the entityId field.

For length, VirtualColumnDefinition is for OAL generated entity, so it is 512 by default. DefinedByField is everywhere, we don't need them always 512, many fields don't need so long.

yazong commented 4 years ago

Your formats are not well. It is very hard to read. For function(s), I think it still missed the length for the entityId field.

For length, VirtualColumnDefinition is for OAL generated entity, so it is 512 by default. DefinedByField is everywhere, we don't need them always 512, many fields don't need so long.

thanks. i understand that i need to read the source code to understand it better.

wu-sheng commented 4 years ago 
PercentileFunction
HistogramFunction
AvgHistogramFunction
AvgFunction

EntityId in these should add 512 length annotation. If you have time, please send a pull request to fix it.

yazong commented 4 years ago 
PercentileFunction
HistogramFunction
AvgHistogramFunction
AvgFunction

EntityId in these should add 512 length annotation. If you have time, please send a pull request to fix it.

https://github.com/apache/skywalking/pull/5057