Balvald / ArtemisScannerTracker

An EDMC plugin that keeps track of the exobiology scanner of a CMDRs Artemis suit in ED:Odyssey
GNU General Public License v3.0
24 stars 4 forks source link

[BUG] Plugin forces EMDC and other plugins to wait while it does its journalcrawling #60

Open Balvald opened 5 months ago

Balvald commented 5 months ago

Describe the bug When using the 'Scan game journals for exobiology' the plugin goes into a long procedure that takes a while. During this time nothing else can be done by EDMC.

To Reproduce Steps to reproduce the behavior:

  1. Make sure there are some game journals available for the plugin to read through.
  2. Go to the 'AST' tab in the EDMC settings.
  3. Click on 'Scan game journals for exobiology' or 'Scan local journals for exobiology'

Expected behavior The journal crawling starts and does not prevent EDMC or other EDMC plugins from working while it reads through journals.

Desktop (please complete the following information):

Additional context Add any other context about the problem here.

Balvald commented 5 months ago

Modified buttons that create a new thread can be found on dev branch they do work though with that arise a few more different potential problems to solve.

What happens if main and newly created thread attempt to access the same json at the same time.

Balvald commented 3 months ago

Technically an issue that if we journal crawl while we play that there is the potential that the plugin will not take notice of a certain event. Though that can be fixed by using the journal crawling again while not playing at the critical point in time.

ultimatespirit commented 3 months ago

Since threads will run into the GIL problem for python (at least until that hopefully goes away in 3.13...) no matter what, it may make sense to spin off a whole other process to do the backlog scanning, and only the backlog scanning, into an intermediate set of json files (keep a log file of what journals were scanned already, if you're paranoid maybe include the hash of the journal file along with its filename if worried about a same-name file being content wise different). Once it finishes it writes to a shared pipe / status file (or just rely on its child exit code) and that's it for inter-process communication between the two.

Your main plugin just continues to do what it was always doing and EDMC doesn't stall, nor does the main plugin potentially miss a just-in-time journal update event from running into threading issues. Only upon that "status all done" cleanup call, or upon a new boot in the event EDMC was closed while a scan was occurring, does it load in the newly parsed data / combine the intermediate data into your main data files.