search

neo4j / neo4j

Graphs for Everyone
http://neo4j.com
GNU General Public License v3.0
13.29k stars 2.38k forks source link

How to resume a stopped or cancelled import #12764

Closed tghdy closed 3 years ago

tghdy commented 3 years ago

A running import was canceled, How to resume it?

  1. There is an error displaied while visitting the database: Database "***" is unavailable, its status is "offline".

  2. I can only find this from neo4j document: https://neo4j.com/docs/operations-manual/current/tools/neo4j-admin-import/#import-tool-resume . But it doesn't help.

  3. It seems neo4j-admin import cann't run twice on the same folder.

    However, using the import command of neo4j-admin is generally faster since it is run against a stopped and empty database.

so, Is there any way ro resume a cancelled import ?

tinwelint commented 3 years ago

What's the error you get? An import should be resumed if tried again, i.e. if it completed importing nodes/relationships, but failed/got cancelled at a later point it should resume from there.

tghdy commented 3 years ago

What's the error you get? An import should be resumed if tried again, i.e. if it completed importing nodes/relationships, but failed/got cancelled at a later point it should resume from there.

Thank you for the responses. My node/edge did not complete the import, and the import was aborted. Can import be resumed in this case?

I tried to use neo4j-admin tool for the second load, the error content is as follows,

WARNING Import failed. The store files in /data/databases/test are left as they are, although they are likely in an unusable state. Starting a database on these store files will likely fail or observe inconsistent records so start at your own risk or delete the store manually

tinwelint commented 3 years ago

What's the error you get? An import should be resumed if tried again, i.e. if it completed importing nodes/relationships, but failed/got cancelled at a later point it should resume from there.

Thank you for the responses. My node/edge did not complete the import, and the import was aborted. Can import be resumed in this case?

It will be restarted from the beginning in that case.

I tried to use neo4j-admin tool for the second load, the error content is as follows,

WARNING Import failed. The store files in /data/databases/test are left as they are, although they are likely in an unusable state. Starting a database on these store files will likely fail or observe inconsistent records so start at your own risk or delete the store manually

There should be more output higher up shouldn't there? That is the last that is printed in case of failure.

tghdy commented 3 years ago

I tried to use neo4j-admin tool for the second load, the error content is as follows, WARNING Import failed. The store files in /data/databases/test are left as they are, although they are likely in an unusable state. Starting a database on these store files will likely fail or observe inconsistent records so start at your own risk or delete the store manually

There should be more output higher up shouldn't there? That is the last that is printed in case of failure.

Yes, the full error log is as follows: Import error: null Caused by:null java.lang.NullPointerException at org.neo4j.internal.batchimport.staging.HumanUnderstandableExecutionMonitor.endPrevious(HumanUnderstandableExecutionMonitor.java:226) at org.neo4j.internal.batchimport.staging.HumanUnderstandableExecutionMonitor.start(HumanUnderstandableExecutionMonitor.java:191) at org.neo4j.internal.batchimport.staging.MultiExecutionMonitor.start(MultiExecutionMonitor.java:64) at org.neo4j.internal.batchimport.staging.ExecutionSupervisor.start(ExecutionSupervisor.java:84) at org.neo4j.internal.batchimport.staging.ExecutionSupervisor.supervise(ExecutionSupervisor.java:62) at org.neo4j.internal.batchimport.staging.ExecutionSupervisors.superviseExecution(ExecutionSupervisors.java:83) at org.neo4j.internal.batchimport.ImportLogic.executeStage(ImportLogic.java:616) at org.neo4j.internal.batchimport.ImportLogic.calculateNodeDegrees(ImportLogic.java:342) at com.neo4j.internal.batchimport.RestartableParallelBatchImporter$2.run(RestartableParallelBatchImporter.java:171) at com.neo4j.internal.batchimport.RestartableParallelBatchImporter.runRemainingStates(RestartableParallelBatchImporter.java:246) at com.neo4j.internal.batchimport.RestartableParallelBatchImporter.doImport(RestartableParallelBatchImporter.java:129) at org.neo4j.importer.CsvImporter.doImport(CsvImporter.java:193) at org.neo4j.importer.CsvImporter.doImport(CsvImporter.java:158) at org.neo4j.importer.ImportCommand.execute(ImportCommand.java:256) at org.neo4j.cli.AbstractCommand.call(AbstractCommand.java:71) at org.neo4j.cli.AbstractCommand.call(AbstractCommand.java:34) at picocli.CommandLine.executeUserObject(CommandLine.java:1953) at picocli.CommandLine.access$1300(CommandLine.java:145) at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) at picocli.CommandLine.execute(CommandLine.java:2078) at org.neo4j.cli.AdminTool.execute(AdminTool.java:89) at org.neo4j.cli.AdminTool.main(AdminTool.java:67) Suppressed: java.lang.NullPointerException at org.neo4j.internal.batchimport.staging.HumanUnderstandableExecutionMonitor.endPrevious(HumanUnderstandableExecutionMonitor.java:226) at org.neo4j.internal.batchimport.staging.HumanUnderstandableExecutionMonitor.done(HumanUnderstandableExecutionMonitor.java:445) at org.neo4j.internal.batchimport.staging.MultiExecutionMonitor.done(MultiExecutionMonitor.java:82) at org.neo4j.internal.batchimport.ImportLogic.close(ImportLogic.java:560) at com.neo4j.internal.batchimport.RestartableParallelBatchImporter.doImport(RestartableParallelBatchImporter.java:118) ... 14 more

WARNING Import failed. The store files in /data/databases/test are left as they are, although they are likely in an unusable state. Starting a database on these store files will likely fail or observe inconsistent records so start at your own risk or delete the store manually java.lang.NullPointerException at org.neo4j.internal.batchimport.staging.HumanUnderstandableExecutionMonitor.endPrevious(HumanUnderstandableExecutionMonitor.java:226) at org.neo4j.internal.batchimport.staging.HumanUnderstandableExecutionMonitor.start(HumanUnderstandableExecutionMonitor.java:191) at org.neo4j.internal.batchimport.staging.MultiExecutionMonitor.start(MultiExecutionMonitor.java:64) at org.neo4j.internal.batchimport.staging.ExecutionSupervisor.start(ExecutionSupervisor.java:84) at org.neo4j.internal.batchimport.staging.ExecutionSupervisor.supervise(ExecutionSupervisor.java:62) at org.neo4j.internal.batchimport.staging.ExecutionSupervisors.superviseExecution(ExecutionSupervisors.java:83) at org.neo4j.internal.batchimport.ImportLogic.executeStage(ImportLogic.java:616) at org.neo4j.internal.batchimport.ImportLogic.calculateNodeDegrees(ImportLogic.java:342) at com.neo4j.internal.batchimport.RestartableParallelBatchImporter$2.run(RestartableParallelBatchImporter.java:171) at com.neo4j.internal.batchimport.RestartableParallelBatchImporter.runRemainingStates(RestartableParallelBatchImporter.java:246) at com.neo4j.internal.batchimport.RestartableParallelBatchImporter.doImport(RestartableParallelBatchImporter.java:129) at org.neo4j.importer.CsvImporter.doImport(CsvImporter.java:193) at org.neo4j.importer.CsvImporter.doImport(CsvImporter.java:158) at org.neo4j.importer.ImportCommand.execute(ImportCommand.java:256) at org.neo4j.cli.AbstractCommand.call(AbstractCommand.java:71) at org.neo4j.cli.AbstractCommand.call(AbstractCommand.java:34) at picocli.CommandLine.executeUserObject(CommandLine.java:1953) at picocli.CommandLine.access$1300(CommandLine.java:145) at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) at picocli.CommandLine.execute(CommandLine.java:2078) at org.neo4j.cli.AdminTool.execute(AdminTool.java:89) at org.neo4j.cli.AdminTool.main(AdminTool.java:67) Suppressed: java.lang.NullPointerException at org.neo4j.internal.batchimport.staging.HumanUnderstandableExecutionMonitor.endPrevious(HumanUnderstandableExecutionMonitor.java:226) at org.neo4j.internal.batchimport.staging.HumanUnderstandableExecutionMonitor.done(HumanUnderstandableExecutionMonitor.java:445) at org.neo4j.internal.batchimport.staging.MultiExecutionMonitor.done(MultiExecutionMonitor.java:82) at org.neo4j.internal.batchimport.ImportLogic.close(ImportLogic.java:560) at com.neo4j.internal.batchimport.RestartableParallelBatchImporter.doImport(RestartableParallelBatchImporter.java:118) ... 14 more

tinwelint commented 3 years ago

Thanks, hmm looks like a bug related to resuming an import, let me try to figure this out.

tinwelint commented 3 years ago

Yup, found the bug. I'll provide a fix for it.

tghdy commented 3 years ago

Thank you for your help. I'll try again later.

tinwelint commented 3 years ago

And of course in the meantime you can simply delete the database directory containing the half-imported data and run it again from the beginning.