treasure-data / digdag

Workload Automation System
https://www.digdag.io/
Apache License 2.0
1.31k stars 221 forks source link

Digdag was crashed due to h2 database full #1636

Closed revanna7226 closed 2 years ago

revanna7226 commented 3 years ago

I set a Digdag with H2 DB and ran for 10 months. Digdag worked fine upto 10 months for our project and crashed, from the available evidences we claimed that digdag is crashed due to database full. When we checked the H2 database tables and data we had created multiple projects in the application. Our intention was to check the diskspace required by database for 1 year. Hence we tried to delete the other projects records from the database. but h2 console is not querying actual data.

SELECT * FROM session_attempts WHERE project_id = 1 is resulting 0 records. But we can records in the table with Project ID is 1 when we view table.

What is happening here? Why this conditional query SELECT * FROM session_attempts WHERE project_id = 1 is not working?

How to check how much diskspace is consuming per session? If we know this we can calculate diskspace required for 1 year.

Thanks for reading... Revanna

hiroyuki-sato commented 3 years ago

Hello, @revanna7226

If digdag projects returns the project which id = 1, It contains project data in your database.

digdag projects
2021-09-06 22:40:30 +0900: Digdag v0.10.2
Projects
  name: hoge
  id: 1
  revision: 273de50f-0967-4838-8de6-5b80f631838b
  archive type: db
  project created at: 2021-09-06 22:40:10 +0900
  revision updated at: 2021-09-06 22:40:10 +0900

Use `digdag workflows <project-name>` to show details.

This ER and this comment may help. (But, I've never tried it. I recommend you to backup the database before try)

68747470733a2f2f71696974612d696d6167652d73746f72652e73332e616d617a6f6e6177732e636f6d2f302f36313436342f64323636333130662d623766302d643733622d383961632d6330323532366363646462322e706e67

digdag_er.pdf

hiroyuki-sato commented 3 years ago

digdag: 0.10.2 h2: h2-1.4.192.jar

digdag.conf

database.type = h2
database.path = ./digdag.db
java -jar /tmp/h2-1.4.192.jar
digdag2 digdag1 digdag3