search

manusimidt / py-xbrl

Python-based parser for parsing XBRL and iXBRL files
https://py-xbrl.readthedocs.io/en/latest/
GNU General Public License v3.0
111 stars 45 forks source link

Solution to frequently missing taxonomy specifications in UK submissions #112

Open manusimidt opened 1 year ago

manusimidt commented 1 year ago

Many UK submissions are missing a proper taxonomy declaration. They just use the prefix without specifying the corresponding namespace and/or do not reference the taxonomy schema file. Currently, the parser is not able to handle those issues, since It does not know where this taxonomy is located.

This issue occurs frequently with the following prefixes:

manusimidt commented 1 year ago

76 is related

twright8 commented 1 year ago

Hi there - apologies for lack of knowledge of this, but for UK data i am frequently unable to pass statements because of:

Was this fixed? I assume its a problem with the UK data.

Thanks very much

manusimidt commented 1 year ago

Hey, no worries. Yes, it's true that these errors still occur often on UK submissions because many companies do not properly define their schema_urls. I searched the internet specifically for this "dpl-frs" taxonomy. However, I could not find an online entry point (only .zip folders containing the taxonomy).

But finding some solution to this problem is in the top 3 list of fixes I want to implement in the next month. Will keep you updated.

twright8 commented 1 year ago

@manusimidt I realise that this is provided for free and its an excellent solution, but i wonder if the UK aspect is any closer to completion? I am close to completing a research report and would ideally like to use this (with attribution!)

manusimidt commented 1 year ago

The issue I still have with UK submissions is that many of them do not properly import their underlying taxonomies. Additionally, the taxonomy endpoints are extremely hard to find and sometimes even not reachable.

If I would find the taxonomy schema files on the web it would be pretty easy to implement.

For example, the taxonomy endpoints for us submissions are pretty straightforward and easy to find: Name Prefix Namespace Schema URL
DEI 2011 dei http://xbrl.sec.gov/dei/2011-01-31 https://xbrl.sec.gov/dei/2011/dei-2011-01-31.xsd
DEI 2012 dei http://xbrl.sec.gov/dei/2012-01-31 https://xbrl.sec.gov/dei/2012/dei-2012-01-31.xsd
DEI 2013 dei http://xbrl.sec.gov/dei/2013-01-31 https://xbrl.sec.gov/dei/2013/dei-2013-01-31.xsd
... ... ... ...
US GAAP 2022 us-gaap http://fasb.org/us-gaap/2022 https://xbrl.fasb.org/us-gaap/2022/elts/us-gaap-2022.xsd
US GAAP 2023 us-gaap http://fasb.org/us-gaap/2023 https://xbrl.fasb.org/us-gaap/2023/elts/us-gaap-2023.xsd
If I had the same table for the UK taxonomies, I could really easy fix this issue. However, I can't find proper taxonomy endpoints for the taxonomies where the parser fails: Name Prefix Namespace Schema URL
?? dpl-countries ?? ??
FRC 2021 core or frc-core http://xbrl.frc.org.uk/fr/2021-01-01/core ??
FRC 2022 core or frc-core http://xbrl.frc.org.uk/fr/2022-01-01/core ??

So basically if I find the missing taxonomy schema files on the web (and basically complete the table for all the years), it would be relatively easy to fix this issue.

twright8 commented 1 year ago

@manusimidt does this page contain what you need? https://www.frc.org.uk/accountants/frc-taxonomies#current-taxonomies-downloads

or

https://uk-taxonomies-tdp.corefiling.com/yeti/resources/yeti-gwt/Yeti.jsp#tax~(id~9*v~150)!net~(a~160*l~32)!lang~(code~en)!rg~(rg~4*p~2)

Realise they're not end points but have the xsd files

manusimidt commented 1 year ago

https://uk-taxonomies-tdp.corefiling.com/yeti/resources/yeti-gwt/Yeti.jsp Is a great way to visualize the taxonomy structure. They also have the FRC 2022 - 2024 Taxonomies, however just visualized for the user and not machine readable.

https://www.frc.org.uk/accountants/frc-taxonomies#current-taxonomies-downloads Is the correct location for downloading the FRC taxonomy. But as far as I can see they only provide .zip folder downloads containing the entire taxonomy and supporting documents. For py-xbrl I ideally would need a direct endpoint (.xds file on server) to easily fetch it. There are basically two workarounds:

  1. Upload the extracted .zip folder to some server and map the namespace to this url
  2. Manually implement functionality to download the .zip folder, extract the necessary documents and map the prefixes, namespaces and (local) schema URLs.

But this would not even solve the full problem, since there site provides only the FRC Taxonomies (as far as I can see). I still don't know to which the other prefixes and namespaces belong.. (i.e.: "dpl-countries", " dpl-frc"...)

twright8 commented 1 year ago

On your last point, I think (but admit i dont know much about this) countries and frc are in the Zip folders, but named slightly differently. But yeah i see your point. How frustrating! I can try and raise this with companies house

Cave-Johnson commented 1 year ago

Was there any progress on this with companies house? Currently running in to the same issue I have some of found the dpl files here : https://www.frc.org.uk/library/standards-codes-policy/accounting-and-reporting/frc-taxonomies/

They are in zip format but could we put them in a public s3 bucket or a dedicated github project and map them to the right namespace?

Cave-Johnson commented 1 year ago

Are these any of the missing Schema URLs?

This also works for previous years ie https://xbrl.frc.org.uk/fr/2022-01-01/core/frc-core-full-2022-01-01.xsd

Cave-Johnson commented 11 months ago

hi @manusimidt - what is needed for me to create a PR with these additional taxonimies? Are you able to link to a similar PR demonstrating whats needed? Happy to have a go at getting these added (and anymore i can find)

manusimidt commented 11 months ago

@Cave-Johnson thanks for researching and providing the links. They indeed look promising! I am not aware of any PR which addressed this problem.

Keep in mind that most of the errors are caused not because the schema URL of the taxonomy is missing but because both the schema URL and the namespace of a given taxonomy are missing. This creates for example the rather frequent error KeyError: 'dpl-frs'.

Here a simple map between the namespace and schema URL is not enough since the namespace dpl-frs could refer to multiple taxonomies. In this case we basically have two options:

  1. Always map the prefix of the taxonomy to the newest taxonomy version that is available and hope for compatibility
  2. Map the prefix of the taxonomy to the most probable taxonomy (i.e.: filing submitted on 31.10.2019 will most probably use the taxonomy version that was released in January 2019).
manusimidt commented 11 months ago

Another option would be that the parser just throws a warning and just ignores all facts that were tagged with the taxonomy which could not be imported. However, this could lead to cascading problems since the parser for example also parses XBRL footnotes. If the fact which the footnote references is not present due to missing taxonomy then the assigning of the footnote to the fact will also fail. This case could be avoided by a simple if-check, but must still be taken into account if such a solution were to be implemented.

manusimidt commented 11 months ago

Probably the last option (to just ignore facts tagged with not locatable taxonomies) would be the best one. In the last two years 90% of my time I invested in py-xbrl was to implement a workaround because some submissions from some country doesn't strictly follow the XBRL standard. However, all these exceptions really bloat the library and make the code harder and harder to read and maintain.

I also tried to formulate this in #84. I hope that I have some time over Christmas to dive into this. In the meantime any PRs or suggestions are highly appreciated :)

manusimidt commented 11 months ago

I just ran a test with new UK submissions and noticed that the schema URL for the following namespace was often missing: 

The taxonomy with namespace **http://www.hmrc.gov.uk/schemas/ct/dpl/2021-01-01** could not be found

@Cave-Johnson already found the DPL taxonomy for 2023: https://xbrl.frc.org.uk/dpl/2023-01-01/dpl-2023-01-01.xsd If we find a 2021 version these errors simply could be resolved for now by adding it to the mapping

Cave-Johnson commented 11 months ago

I'll do some digging and see what I can find

Brief search this might be a good source for UK accepted taxonomies

https://www.gov.uk/government/publications/taxonomies-accepted-by-hm-revenue-and-customs/taxonomies-accepted-by-hmrc

On Tue, 21 Nov 2023, 22:21 Manuel Schmidt, @.***> wrote:

I just ran a test with new UK submissions and noticed that the schema URL for the following namespace was often missing:

The taxonomy with namespace http://www.hmrc.gov.uk/schemas/ct/dpl/2021-01-01 could not be found

@Cave-Johnson https://github.com/Cave-Johnson already found the DPL taxonomy for 2023: https://xbrl.frc.org.uk/dpl/2023-01-01/dpl-2023-01-01.xsd If we find a 2021 version these errors simply could be resolved for now by adding it to the mapping https://github.com/manusimidt/py-xbrl/blob/485a428e74425d7bbdd5aa632d0310955166e6f6/xbrl/taxonomy.py#L31

— Reply to this email directly, view it on GitHub https://github.com/manusimidt/py-xbrl/issues/112#issuecomment-1821793954, or unsubscribe https://github.com/notifications/unsubscribe-auth/AET5RUHWSEUQDB7MWFXS7EDYFUSOBAVCNFSM6AAAAAAYITI6VCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRRG44TGOJVGQ . You are receiving this because you were mentioned.Message ID: @.***>

Cave-Johnson commented 11 months ago

A bit more digging i have found the following https://www.hmrc.gov.uk/schemas/ct/combined/2021-01-01/dpl-2021-frs-102-2021-v1.0.0.xsd which lead me to

Are these what was missing?

keyboardcrimp commented 10 months ago

Been having the same issue recently when parsing companies house docs, was wondering if there was any update on this please?

manusimidt commented 10 months ago

I have tried it with the taxonomies provided by @Cave-Johnson, however, unfortunately, they did not work for all of the failed submissions I tested on. The instance documents were using tags like dpl-countries:CountriesDimension which are not present in the taxonomy schema.

manusimidt commented 10 months ago

I think the only viable option is to build in some "fail gracefully" mode into py-xbrl in which the parser returns a partially parsed xbrl instance document and just omits every fact/context that is labeled/uses a concept from a taxonomy which could not be located.

Cave-Johnson commented 8 months ago

understand your point RE fail gracefully logic and that makes sense. It would still be good to get the taxonomies linked where possible.

Ive done a bit more digging and come across the attached that references dpl countries in the dpl-2021-presentation.xml file - is this of any use? DPL-2021-v1.1.0.zip

twright8 commented 3 months ago

@manusimidt Thanks for all the looking into this! was just wondering if @Cave-Johnson solution could have helped?