search

cubefs / compass

Compass is a task diagnosis platform for bigdata
Apache License 2.0
358 stars 133 forks source link

[Question]: Why after running Spark task, there is no change of the database? (both on postgresql and mysql) #216

Open huanghuihui0904 opened 6 months ago

huanghuihui0904 commented 6 months ago

Contact Details

huihuihuang0904@gmail.com

What would you like to ask or discuss?

After I ran the Spark task by the instructions you provide, there is no change of the database. Can you tell me how to solve that? Also I wonder how the program write the data to the database, in which way or in which java files? As I have checked the error logs of my program, it just has the following two errors which I think not the key point. In /opt/compass-v1.1.2/task-canal/logs image In /opt/compass-v1.1.2/task-portal/logs
image Look forward to your reply!

huanghuihui0904 commented 6 months ago

I want to add that now I fill all the table in mysql database(except blocklist), but the webpage still has no data. I add the data by the following sqls:

INSERT INTO task_application VALUES (1, 'application_1712475671665_0002', 'Spark Pi', 'Spark Pi', 'Spark Pi', 'Spark', '2024-04-07 16:22:47', 1, 'http://192.168.106.192:18018/api/v1/applications/application_1712475671665_0001/1/logs', '2024-04-07 16:22:47', '2024-04-07 16:23:01'); INSERT INTO project VALUES (1, 'Spark Pi', 'description', 1, 1, '2024-04-07 16:22:47','2024-04-07 16:23:01'); INSERT INTO task VALUES (1,'Spark Pi',1,'Spark Pi',1,'Spark Pi','description',1,'SPARK',2,'2024-04-07 16:22:47', '2024-04-07 16:23:01'); INSERT INTO flow VALUES (1,'Spark Pi','description',1,1,1,'Spark Pi','2024-04-07 16:22:47', '2024-04-07 16:23:01'); INSERT INTO template VALUES(1,1,'cluster','advice','log','2024-04-07 16:23:01','2024-04-07 16:22:47') INSERT INTO task_instance VALUES(1,'Spark Pi','Spark Pi','Spark Pi','2024-04-07 16:23:01','2024-04-07 16:23:01','2024-04-07 16:23:01.000000','FINISHED','SPARK',4,5,'worker group','schedule','2024-04-07 16:23:01','2024-04-07 16:23:01') INSERT INTO task_datum VALUES(1,'Spark Pi','Spark Pi','Spark Pi','2024-04-07 16:23:01','baseline','2024-04-07 16:22:47') INSERT INTO flink_task VALUES(1,'Spark Pi',1,'Spark Pi',1,'Spark Pi',1,0,'2024-04-07 16:22:47','2024-04-07 16:23:01') INSERT INTO flink_task_app VALUES(1,'compass',1,'Spark Pi',1,'Spark Pi',1,'Spark Pi',1,'finish',1,'2024-04-07 16:23:01',1,'flink url',20,30,40,'Hadoop',100,'2024-04-07 16:22:47','2024-04-07 16:24:01',60,40,'queue','cluster',9,'compass','diagnosis',2,3,4,6,7,'Spark Pi','2024-04-07 16:22:47','2024-04-07 16:23:01')

Addtionally, I try the url http://192.****:7075/compass/api/v1/report/projects and the right response, which means there is no error in the mysql connection. But the table of database dolphinscheduler is empty, does this lead to the empty webpage?

nilnon commented 6 months ago

Contact Details

huihuihuang0904@gmail.com

What would you like to ask or discuss?

After I ran the Spark task by the instructions you provide, there is no change of the database. Can you tell me how to solve that? Also I wonder how the program write the data to the database, in which way or in which java files? As I have checked the error logs of my program, it just has the following two errors which I think not the key point. In /opt/compass-v1.1.2/task-canal/logs image In /opt/compass-v1.1.2/task-portal/logs image Look forward to your reply!

@huanghuihui0904 Check whether the canal dependency package has been downloaded?

nilnon commented 6 months ago

I want to add that now I fill all the table in mysql database(except blocklist), but the webpage still has no data. I add the data by the following sqls:

INSERT INTO task_application VALUES (1, 'application_1712475671665_0002', 'Spark Pi', 'Spark Pi', 'Spark Pi', 'Spark', '2024-04-07 16:22:47', 1, 'http://192.168.106.192:18018/api/v1/applications/application_1712475671665_0001/1/logs', '2024-04-07 16:22:47', '2024-04-07 16:23:01'); INSERT INTO project VALUES (1, 'Spark Pi', 'description', 1, 1, '2024-04-07 16:22:47','2024-04-07 16:23:01'); INSERT INTO task VALUES (1,'Spark Pi',1,'Spark Pi',1,'Spark Pi','description',1,'SPARK',2,'2024-04-07 16:22:47', '2024-04-07 16:23:01'); INSERT INTO flow VALUES (1,'Spark Pi','description',1,1,1,'Spark Pi','2024-04-07 16:22:47', '2024-04-07 16:23:01'); INSERT INTO template VALUES(1,1,'cluster','advice','log','2024-04-07 16:23:01','2024-04-07 16:22:47') INSERT INTO task_instance VALUES(1,'Spark Pi','Spark Pi','Spark Pi','2024-04-07 16:23:01','2024-04-07 16:23:01','2024-04-07 16:23:01.000000','FINISHED','SPARK',4,5,'worker group','schedule','2024-04-07 16:23:01','2024-04-07 16:23:01') INSERT INTO task_datum VALUES(1,'Spark Pi','Spark Pi','Spark Pi','2024-04-07 16:23:01','baseline','2024-04-07 16:22:47') INSERT INTO flink_task VALUES(1,'Spark Pi',1,'Spark Pi',1,'Spark Pi',1,0,'2024-04-07 16:22:47','2024-04-07 16:23:01') INSERT INTO flink_task_app VALUES(1,'compass',1,'Spark Pi',1,'Spark Pi',1,'Spark Pi',1,'finish',1,'2024-04-07 16:23:01',1,'flink url',20,30,40,'Hadoop',100,'2024-04-07 16:22:47','2024-04-07 16:24:01',60,40,'queue','cluster',9,'compass','diagnosis',2,3,4,6,7,'Spark Pi','2024-04-07 16:22:47','2024-04-07 16:23:01')

Addtionally, I try the url http://192.****:7075/compass/api/v1/report/projects and the right response, which means there is no error in the mysql connection. But the table of database dolphinscheduler is empty, does this lead to the empty webpage?

@huanghuihui0904 Part of the data of webpage comes from mysql, and part of it comes from elasticsearch/opensearch.

nilnon commented 6 months ago

@huanghuihui0904 You need to run all modules successfully and run spark tasks in dolphinscheduler so that task-portal can display data.

huanghuihui0904 commented 6 months ago

But my opensearch can work well as I can open the web and see the data. So do you mean the error with canal can lead to the problem: (1) after running spark task, no data written to mysql. (2)the webpage(actually the request http://192.*****:7075/compass/api/v1/report/statistics?projectName=) cannot get data. Add some info to the request http://192.*****:7075/compass/api/v1/report/statistics?projectName=: The responce as the following.

{ "code": 200, "msg": "Succeed", "data": null } And I find that this is the main request to get the data and then let the webpage show.

huanghuihui0904 commented 6 months ago

@huanghuihui0904 You need to run all modules successfully and run spark tasks in dolphinscheduler so that task-portal can display data.

Do you mean the dolphinscheduler database? As I just fill the data in the compass database.

nilnon commented 6 months ago

@huanghuihui0904 You need to run all modules successfully and run spark tasks in dolphinscheduler so that task-portal can display data.

Do you mean the dolphinscheduler database? As I just fill the data in the compass database.

But my opensearch can work well as I can open the web and see the data. So do you mean the error with canal can lead to the problem: (1) after running spark task, no data written to mysql. (2)the webpage(actually the request http://192.*****:7075/compass/api/v1/report/statistics?projectName=) cannot get data. Add some info to the request http://192.*****:7075/compass/api/v1/report/statistics?projectName=: The responce as the following.

{ "code": 200, "msg": "Succeed", "data": null } And I find that this is the main request to get the data and then let the webpage show.

the /api/v1/report/statistics data mainly come from opensearch.