hlageek
commented
3 years ago Tohle právě umí grobid, který je stále under active development, takže myslím, že se stále zlepšuje. Byť na jiných než vědeckých článcích ty výsledky asi nebudou tak dobré. Nicméně rozlišení strukturních prvků - jako co je text a co je paratext - by mohlo dobře fungovat i na jiných vstupech.
Online demo grobidu je zde: https://cloud.science-miner.com/grobid/
Zde je příklad výstupu, z nějž by stačilo z elementu text extrahovat element p, aby člověk získal pouze text dokumentu.
<?xml version="1.0" encoding="UTF-8"?>
<TEI xml:space="preserve"
xmlns="http://www.tei-c.org/ns/1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.tei-c.org/ns/1.0 https://raw.githubusercontent.com/kermitt2/grobid/master/grobid-home/schemas/xsd/Grobid.xsd"
xmlns:xlink="http://www.w3.org/1999/xlink">
<teiHeader xml:lang="en">
<fileDesc>
<titleStmt>
<title level="a" type="main">This form is to be used for all personnel who do not hold an Australian Government national security clearance, including ongoing and non-ongoing employees, contractors, service providers and Attached Agency staff to report changes in their circumstances to the Austrade Security Team. If you hold an Australian Government Security Clearance, please do not use this form. Please complete the SVA 003 Change of Circumstances Notification Form that can be found on the Personnel Security HUB Page on the Hub and provide to the Austrade Security Team when completed. To: Austrade Security Team</title>
</titleStmt>
<publicationStmt>
<publisher/>
<availability status="unknown">
<licence/>
</availability>
</publicationStmt>
<sourceDesc>
<biblStruct>
<analytic>
<title level="a" type="main">This form is to be used for all personnel who do not hold an Australian Government national security clearance, including ongoing and non-ongoing employees, contractors, service providers and Attached Agency staff to report changes in their circumstances to the Austrade Security Team. If you hold an Australian Government Security Clearance, please do not use this form. Please complete the SVA 003 Change of Circumstances Notification Form that can be found on the Personnel Security HUB Page on the Hub and provide to the Austrade Security Team when completed. To: Austrade Security Team</title>
</analytic>
<monogr>
<imprint>
<date/>
</imprint>
</monogr>
</biblStruct>
</sourceDesc>
</fileDesc>
<encodingDesc>
<appInfo>
<application version="0.6.2-SNAPSHOT" ident="GROBID" when="2021-02-02T10:33+0000">
<desc>GROBID - A machine learning software for extracting information from scholarly documents</desc>
<ref target="https://github.com/kermitt2/grobid"/>
</application>
</appInfo>
</encodingDesc>
<profileDesc>
<abstract/>
</profileDesc>
</teiHeader>
<text xml:lang="en">
<body>
<div
xmlns="http://www.tei-c.org/ns/1.0">
<p>(dd/mm/yyyy)</p>
</div>
<div
xmlns="http://www.tei-c.org/ns/1.0">
<head>Change of Circumstances Notification</head>
</div>
<div
xmlns="http://www.tei-c.org/ns/1.0">
<head>Non-Security Clearance Holders (OEE and A-Based Staff)</head>
<p>OFFICIAL: Sensitive (after first entry) </p>
</div>
<div
xmlns="http://www.tei-c.org/ns/1.0">
<head>Section 3 -Declaration and Signature</head>
<p>I hereby declare that to the best of my knowledge and belief, the information provided in this form is true and complete. The supporting documents I have provided relate to me, are complete and correct and are issued by the issuing authority without alterations.</p>
<p>Date: Signature:</p>
<p>OFFICIAL: Sensitive (after first entry)</p>
</div>
<div
xmlns="http://www.tei-c.org/ns/1.0">
<head>Change of Circumstances Information</head>
<p>The following list is a comprehensive, but not exhaustive, list of changes that must be reported to the Austrade Vetting Team: includes any contact with government officials that is not work-related or required to fulfil the responsibilities of your position with Austrade. This also includes work-related contact that you find suspicious; f) A family member, spouse/partner or person that you live with, being employed by a government organisation/agency (other than the Australian Government): please provide details of the organisation/agency, position title and period of employment; g) Changes in citizenship or nationality: If you assume a new citizenship, by either renouncing your current citizenship or attaining dual citizenship, the Austrade Vetting Team must be advised. Please also provide a certified copy of your new citizenship certificate to the Austrade Vetting Team; h) Involvement in criminal activity: this includes deliberate and accidental involvement; i) Criminal charges, warning or convictions; j) Involvement with any individual, group, society or organisation that may be of security concern or present a conflict of interest: this may include special interest groups that you are involved in that are not work-related or required to fulfil the responsibilities of your position with Austrade. This includes joining a political party; k) Change of employment within Austrade: position or location; l) Employment outside of Austrade: please provide dates of employment, organisation/agency, position title and reason for leaving position (if applicable). You will need to provide the Austrade Vetting Team with evidence of your employment i.e. payslip, letter of offer, contract of employment etc.; m) Security incidents: this includes security breaches and agency or employer sanctions for code of conduct misconduct; n) New passport/s: please provide a copy of any new passport/s which have been issued since your Pre-Employment Screening Pack was completed or last reviewed and have not previously been provided to the Austrade Vetting Team; o) Changes in financial circumstances: this may include receipt or giving of large amounts of money, significant increases in debt or possible financial hardship and bankruptcy. Generally this equates to plus or minus $10,000 AUD in each instance; p) Any other change in your circumstances that may be of concern to Austrade.</p>
</div>
<figure
xmlns="http://www.tei-c.org/ns/1.0" xml:id="fig_0">
<head>Section 1 :</head>
<label>1</label>
<figDesc>Your detailsSection 2: Provide details of the change of circumstance. Further guidance is located on page 3 of this form. Section 3: To be signed by you</figDesc>
</figure>
daliboris
bodnarIQ
stranak
bukovskyIQ
Požiadavka: "Ve výzkumu mě zajímá samotný text odborných článků, měl bych mít tedy možnost z textu odfiltrovat paratext (čísla stránek a záhlaví s nadpisy apod.)"
Na toto sa mi nepodarilo nájsť žiaden nástroj, ktorý by to dokázal vyriešiť z textového reťazca. Bolo by to ale možné vyriešiť použitím lepších nástrojov pre OCR, ktoré by rozpoznaný text vrátili otagovaný, pomocou čoho by sme sa mohli rozhodovať, že či ide o paratext alebo nie. Verzovanie stránok podľa použitého OCR máme vyriešené, odfiltrovanie paratextu by ale tým pádom fungovalo iba na novších OCR. Máte nejaké dostupné nástroje na OCR, ktoré by tohoto boli schopné, alebo budeme musieť nejaký nástroj nájsť? Zatiaľ som narazil na Abby FineReader, ktorý sa najčastejšie považuje za najlepší.
Ďaľším potencionálnym riešením by bolo použitie REGEXu, čo ale nebude veľmi spoľahlivé, keďže obsah, na ktorom by sme to chceli použiť, je veľmi rozmanitý(niekedy je číslovanie stránok na začiatku, niekedy na konci, mohlo by dochádzať k odstráneniu časti textu ktorý nie je paratextom)