search

keeps / dbptk-ui

DBPTK base UI for both Desktop and Enterprise
https://database-preservation.com
GNU Lesser General Public License v3.0
23 stars 9 forks source link

dbptk-desktop runs out of space, but not telling where #312

Open Tarjei-stavanger opened 2 years ago

Tarjei-stavanger commented 2 years ago

Description:

I understand that the database I’m trying to archive is rather large or complicated. Or both.

Dbptk-desktop still has the same problem running out of storage with temp on the 12TB disk.

However, the real problem is it is impossible to know from the JVM log file where it is running out of storage. Or if it is just a numeric overflow because a file is larger than a 32-bit integer can measure.

An error message like the following is not terribly informative : 2022-02-23 15:56:14.276 ERROR 15540 --- [o-auto-1-exec-1] o.a.solr.handler.RequestHandlerBase : org.apache.solr.common.SolrException: Server error writing document id 0e09c29d-0f6d-4e79-9665-cd1fa8de301d to the index

Context:

Steps required to reproduce the bug:

Export Microsoft SQL database. Using integrated login and disable encryption Apply Merkle Tree Filter with defaults SIARD Export Options Compress

Attach any related screenshots below.

Attach any relevant logs below.

Find DBPTK Desktop logs at Help -> Logs menu.

Shortened the jvm.log from 1.1GB to 617KB.

jvm.log

Tarjei-stavanger commented 2 years ago

I am now regularly checking space use during the convert job and notice that the dbptk-desktop is using the users Temp area despite being directed to use a 12TB drive as temp.

It appears that the only item that respects the setting of the TEMP environment variable is the placement of the jvm.log.

No wonder it runs out of space when it is still using C:.

image

luis100 commented 2 years ago

@Tarjei-stavanger have you set the Temporary directory setting as described in https://github.com/keeps/dbptk-ui/issues/311#issuecomment-1048658968?

Tarjei-stavanger commented 2 years ago

I have set it using the TEMP variable and verified that the setting in dbptk-desktop is correct.

Is that not enough ?

From: Luis Faria @.> Sent: Thursday, February 24, 2022 11:07 To: keeps/dbptk-ui @.> Cc: Tarjei Jensen @.>; Mention @.> Subject: Re: [keeps/dbptk-ui] dbptk-desktop runs out of space, but not telling where (Issue #312)

@Tarjei-stavangerhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FTarjei-stavanger&data=04%7C01%7Ctarjei.jensen%40stavanger.kommune.no%7C71657f085ea44e50cdbc08d9f77d7c0e%7C8cb94f174b6045d6b279cd88076e5b48%7C0%7C0%7C637812940605179477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=3yLfYWqDm5afIEMC3mFSjfKTa2QWhQ%2Fn4Laqwsl1xkM%3D&reserved=0 have you set the Temporary directory setting as described in #311 (comment)https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkeeps%2Fdbptk-ui%2Fissues%2F311%23issuecomment-1048658968&data=04%7C01%7Ctarjei.jensen%40stavanger.kommune.no%7C71657f085ea44e50cdbc08d9f77d7c0e%7C8cb94f174b6045d6b279cd88076e5b48%7C0%7C0%7C637812940605179477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=5zuIidllWrbffb2LJD2Uv779%2Byq91kIQtcQZyjS%2BEvE%3D&reserved=0?

- Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkeeps%2Fdbptk-ui%2Fissues%2F312%23issuecomment-1049689944&data=04%7C01%7Ctarjei.jensen%40stavanger.kommune.no%7C71657f085ea44e50cdbc08d9f77d7c0e%7C8cb94f174b6045d6b279cd88076e5b48%7C0%7C0%7C637812940605179477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=vAkZZ%2BacbuYmNFpMrXzqzds8xIiOGk9IZJZsYWa0nuM%3D&reserved=0, or unsubscribehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FATVQGCXKSA4KB754SUNTZGDU4X7NZANCNFSM5PGQOPJA&data=04%7C01%7Ctarjei.jensen%40stavanger.kommune.no%7C71657f085ea44e50cdbc08d9f77d7c0e%7C8cb94f174b6045d6b279cd88076e5b48%7C0%7C0%7C637812940605179477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=uhl%2FrSn4w5w%2FOhIP%2B6gPSymTVHjHrwO3KFbVrVZW5RY%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Ctarjei.jensen%40stavanger.kommune.no%7C71657f085ea44e50cdbc08d9f77d7c0e%7C8cb94f174b6045d6b279cd88076e5b48%7C0%7C0%7C637812940605179477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=vRGQX8lL2hdY0I4iw2l6%2FUwP%2B64EtXfhz2sBxwxxjGw%3D&reserved=0 or Androidhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Ctarjei.jensen%40stavanger.kommune.no%7C71657f085ea44e50cdbc08d9f77d7c0e%7C8cb94f174b6045d6b279cd88076e5b48%7C0%7C0%7C637812940605179477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qC7YcTx6v%2FL2mWaUsr24XP8wvOwLBr6HpKXEQZ2U73w%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.**@.>>

luis100 commented 2 years ago

I'm not sure what you mean with "set it using the TEMP variable", as the DBPTK Desktop spawns a Java process within the Electron app and I don't think it would pickup environment variables set in the context that is initiating the Electron app.

The procedure we have defined is to change the Temporary Directory using the Settings panel. You say you have verified the temporary directory is correct using the Setting panel, but this strategy your are using to set it may be creating the issue of temporary directory misconfiguration and the Setting may reporting it incorrectly. I suggest you remove the previous approach and only set it using the Settings panel.

If, using the recommended approach, you are still reporting the temporary files are still being created in the wrong folder, we can analyse it as a bug on the settings definition.

Tarjei-stavanger commented 2 years ago

I have checked that the TEMP variable is preserved from process to process by setting it in CMD and verifying it in PowerShell and another CMD session.

This is why the content of the variable is picked up by the dbptk-desktop program. If it didn't, then the jvm.log would be on C:

Windows is not Linux where variables have to be exported to be preserved.

From: Luis Faria @.> Sent: Thursday, February 24, 2022 12:29 To: keeps/dbptk-ui @.> Cc: Tarjei Jensen @.>; Mention @.> Subject: Re: [keeps/dbptk-ui] dbptk-desktop runs out of space, but not telling where (Issue #312)

I'm not sure what you mean with "set it using the TEMP variable", as the DBPTK Desktop spawns a Java process within the Electron app and I don't think it would pickup environment variables set in the context that is initiating the Electron app.

The procedure we have defined is to change the Temporary Directory using the Settings panel. You say you have verified the temporary directory is correct using the Setting panel, but this strategy your are using to set it may be creating the issue of temporary directory misconfiguration and the Setting may reporting it incorrectly. I suggest you remove the previous approach and only set it using the Settings panel.

If, using the recommended approach, you are still reporting the temporary files are still being created in the wrong folder, we can analyse it as a bug on the settings definition.

- Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkeeps%2Fdbptk-ui%2Fissues%2F312%23issuecomment-1049764128&data=04%7C01%7Ctarjei.jensen%40stavanger.kommune.no%7Cf50282e952f048cd8ebf08d9f788dde4%7C8cb94f174b6045d6b279cd88076e5b48%7C0%7C0%7C637812989481399889%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=uZ5zqv3aEn9PfTZiGw%2F8I8%2FtiQINENaObm3G%2FCEcKWQ%3D&reserved=0, or unsubscribehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FATVQGCTFCT2XDPTPF7TNZCDU4YI75ANCNFSM5PGQOPJA&data=04%7C01%7Ctarjei.jensen%40stavanger.kommune.no%7Cf50282e952f048cd8ebf08d9f788dde4%7C8cb94f174b6045d6b279cd88076e5b48%7C0%7C0%7C637812989481399889%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=plnFhGStrOzVgDCaJxpT6uq7Y8KqqxG0HHrmgyvDP2w%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Ctarjei.jensen%40stavanger.kommune.no%7Cf50282e952f048cd8ebf08d9f788dde4%7C8cb94f174b6045d6b279cd88076e5b48%7C0%7C0%7C637812989481399889%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=IphEy5RFrj3Hxu%2FGSvOBWPCysjscehY3pkArF56wHQI%3D&reserved=0 or Androidhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Ctarjei.jensen%40stavanger.kommune.no%7Cf50282e952f048cd8ebf08d9f788dde4%7C8cb94f174b6045d6b279cd88076e5b48%7C0%7C0%7C637812989481399889%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=6DniGd4s5A70nlqh1qDErDRZHNLP%2B2Th5L4aRomaVos%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.**@.>>

Tarjei-stavanger commented 2 years ago

When I use your method, the program stops with an error because of an undocumented setting.

The setting is as follows {"disableTimezone":true,"maxHeapMemory":"268435456","language":"en","tmpDir":"D:\Temp"}

From: Luis Faria @.> Sent: Thursday, February 24, 2022 12:29 To: keeps/dbptk-ui @.> Cc: Tarjei Jensen @.>; Mention @.> Subject: Re: [keeps/dbptk-ui] dbptk-desktop runs out of space, but not telling where (Issue #312)

I'm not sure what you mean with "set it using the TEMP variable", as the DBPTK Desktop spawns a Java process within the Electron app and I don't think it would pickup environment variables set in the context that is initiating the Electron app.

The procedure we have defined is to change the Temporary Directory using the Settings panel. You say you have verified the temporary directory is correct using the Setting panel, but this strategy your are using to set it may be creating the issue of temporary directory misconfiguration and the Setting may reporting it incorrectly. I suggest you remove the previous approach and only set it using the Settings panel.

If, using the recommended approach, you are still reporting the temporary files are still being created in the wrong folder, we can analyse it as a bug on the settings definition.

- Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkeeps%2Fdbptk-ui%2Fissues%2F312%23issuecomment-1049764128&data=04%7C01%7Ctarjei.jensen%40stavanger.kommune.no%7Cf50282e952f048cd8ebf08d9f788dde4%7C8cb94f174b6045d6b279cd88076e5b48%7C0%7C0%7C637812989481399889%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=uZ5zqv3aEn9PfTZiGw%2F8I8%2FtiQINENaObm3G%2FCEcKWQ%3D&reserved=0, or unsubscribehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FATVQGCTFCT2XDPTPF7TNZCDU4YI75ANCNFSM5PGQOPJA&data=04%7C01%7Ctarjei.jensen%40stavanger.kommune.no%7Cf50282e952f048cd8ebf08d9f788dde4%7C8cb94f174b6045d6b279cd88076e5b48%7C0%7C0%7C637812989481399889%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=plnFhGStrOzVgDCaJxpT6uq7Y8KqqxG0HHrmgyvDP2w%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Ctarjei.jensen%40stavanger.kommune.no%7Cf50282e952f048cd8ebf08d9f788dde4%7C8cb94f174b6045d6b279cd88076e5b48%7C0%7C0%7C637812989481399889%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=IphEy5RFrj3Hxu%2FGSvOBWPCysjscehY3pkArF56wHQI%3D&reserved=0 or Androidhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Ctarjei.jensen%40stavanger.kommune.no%7Cf50282e952f048cd8ebf08d9f788dde4%7C8cb94f174b6045d6b279cd88076e5b48%7C0%7C0%7C637812989481399889%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=6DniGd4s5A70nlqh1qDErDRZHNLP%2B2Th5L4aRomaVos%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.**@.>>

Tarjei-stavanger commented 2 years ago

Removing the maxHeapMemory parameter from the settings file allows the program to proceed. It may be that setting the TMP variable before starting the program also influences where files are placed.

luis100 commented 2 years ago

It may, if you can reproduce the issue by reseting your configurations and trying to set them again. If your database is very large and you want to streamline your resources and have more control over the application, I would recommend you use the dbptk-developer, on which you can also set the temporary directory.

https://github.com/keeps/dbptk-developer/wiki/Troubleshooting#main-hard-drive-gets-full-due-to-temporary-files