Priority Manager - Githubissues

UP2040499 / auto-osint-v

An automated tool for Validating OSINT. This forms part of the final step of OSINT production as detailed by NATO's open source handbook (2001). This is a research artefact for my Dissertation at the University of Portsmouth.

https://up2040499.github.io/auto-osint-v/

Creative Commons Zero v1.0 Universal

2 stars 0 forks source link

Priority Manager #20

Closed UP2040499 closed 1 year ago

UP2040499 commented 1 year ago

This takes information from Popular Entity Store #19, Target Information Stores #15, and Potential Corroboration Store #23.

Assigns scores for sources that independently mention ‘target information’ and relatively lower scores to sources that proffer ‘popular information’.

Each source gets points for independent mentions, i.e. a source that repeats the given information will only get points for the first mention.

Good (or neutral semantics) semantic analysis results lead to an increase in the source’s overall score. Bad (or emotional semantics) semantic analysis results give the source no points.

Sources are added to Evidence Store #22 with an associated priority score.

UP2040499 commented 1 year ago

For better performance, will want to reduce the number of sources before finding popular entities. This can be done by performing the scoring of sources based on target information first, then taking the top x% of scored sources and putting them into the popular info finder. Finally, can score the remaining sources based on popular information.

UP2040499 commented 1 year ago

So for unknown reasons, when running PopularInformationFinder from within the unit_test works fine. However, when it is run from inside the PriorityManager class, nothing is generated with the multiprocessing.Pool.map function.

UP2040499 commented 1 year ago

To finish:

Fix the error message 'XMLParsedAsHTML' by switching the parser used in BS4, based on the website file type.
Attempt to minimise the 'Token indices sequence length is longer than the specified maximum sequence length for this model.' warning.
- This can be reduced by changing the 'window' and 'stride' in the config of the model
- See Spacy Docs - strided_spans.v1

UP2040499 commented 1 year ago

[x] Fix 'XMLParsedAsHTML' warning
[x] Reduce the 'Token indices' warning occurrence by modifying the window and stride fields in the model config.