Closed EricForgy closed 4 years ago
Hi 👋
Over on Slack, @kescobo was helping me and came up with a good MWE:
julia> test = parsexml("""
<?xml version="1.0" encoding="utf-8"?>
<xbrl
xmlns="http://www.xbrl.org/2003/instance"
xmlns:country="http://xbrl.sec.gov/country/2017-01-31"
xmlns:dei="http://xbrl.sec.gov/dei/2019-01-31"
xmlns:iso4217="http://www.xbrl.org/2003/iso4217"
xmlns:link="http://www.xbrl.org/2003/linkbase"
xmlns:mlic="http://www.metlife.com/20191231"
xmlns:srt="http://fasb.org/srt/2019-01-31"
xmlns:us-gaap="http://fasb.org/us-gaap/2019-01-31"
xmlns:xbrldi="http://xbrl.org/2006/xbrldi"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
</xbrl>""")
EzXML.Document(EzXML.Node(<DOCUMENT_NODE@0x0000000040b48a20>))
julia> findall("/xbrl", test)
0-element Array{EzXML.Node,1}
julia> test2 = parsexml("""
<?xml version="1.0" encoding="utf-8"?>
<xbrl></xbrl>""")
EzXML.Document(EzXML.Node(<DOCUMENT_NODE@0x0000000040b48360>))
julia> findall("/xbrl", test2)
1-element Array{EzXML.Node,1}:
EzXML.Node(<ELEMENT_NODE[xbrl]@0x0000000042f60090>)
As far as I can tell, the former should be a valid XML document.
Any idea what is going on?
~Is it possible that this https://github.com/bicycle1885/EzXML.jl/blob/master/src/xpath.jl#L43~
function Base.findall(xpath::AbstractString, doc::Document)
return findall(xpath, doc.node)
end
~should be~
function Base.findall(xpath::AbstractString, doc::Document)
return findall(xpath, doc.root) # i.e. doc.root instead of doc.node
end
~?~
Edit: Nevermind.
A more minimal MWE:
julia> test = parsexml("""
<?xml version="1.0" encoding="utf-8"?>
<xbrl xmlns="http://www.xbrl.org/2003/instance">
</xbrl>""")
EzXML.Document(EzXML.Node(<DOCUMENT_NODE@0x00007f9d9993d700>))
julia> findall("/xbrl", test)
0-element Array{EzXML.Node,1}
Thanks @kescobo 🙌
I think this might be part of the problem:
julia> test = parsexml("""
<?xml version="1.0" encoding="utf-8"?>
<xbrl xmlns="http://www.xbrl.org/2003/instance">
</xbrl>""")
EzXML.Document(EzXML.Node(<DOCUMENT_NODE@0x0000000008298810>))
julia> namespaces(root(test))
1-element Array{Pair{String,String},1}:
"" => "http://www.xbrl.org/2003/instance"
It seems EzXML does not like the empty key 🤔
Just for fun:
julia> using EzXML
julia> test = parsexml("""
<?xml version="1.0" encoding="utf-8"?>
<primates xmlns="http://www.xbrl.org/2003/instance">
</primates>""")
EzXML.Document(EzXML.Node(<DOCUMENT_NODE@0x00007fa367862de0>))
julia> findall("/primates", test)
0-element Array{EzXML.Node,1}
julia> test2 = parsexml("""
<?xml version="1.0" encoding="utf-8"?>
<primates test="test">
</primates>""")
EzXML.Document(EzXML.Node(<DOCUMENT_NODE@0x00007fa36b074c90>))
julia> findall("/primates", test2)
1-element Array{EzXML.Node,1}:
EzXML.Node(<ELEMENT_NODE[primates]@0x00007fa36b1137f0>)
julia> test3 = parsexml("""
<?xml version="1.0" encoding="utf-8"?>
<xmlns test="test">
</xmlns>""")
EzXML.Document(EzXML.Node(<DOCUMENT_NODE@0x00007fa36b04d610>))
julia> findall("/xmlns", test3)
1-element Array{EzXML.Node,1}:
EzXML.Node(<ELEMENT_NODE[xmlns]@0x00007fa3665fed60>)
julia> test4 = parsexml("""
<?xml version="1.0" encoding="utf-8"?>
<xbrl xmlns="http">
</xbrl>""")
EzXML.Document(EzXML.Node(<DOCUMENT_NODE@0x00007fa36789e170>))
julia> findall("/xbrl", test4)
0-element Array{EzXML.Node,1}
I tried modifying findall(xpath, doc)
, which just calls findall(xpath, doc.node, namespaces(doc.node))
to
function Base.findall(xpath::AbstractString, doc::Document, ns=namespaces(doc.node))
return findall(xpath, doc.node, ns)
end
and then tried
julia> findall("/xbrl", test, namespaces(root(test)))
┌ Warning: ignored the empty prefix for 'http://www.xbrl.org/2003/instance'; expected to be non-empty
└ @ EzXML C:\Users\ericf\.julia\dev\EzXML\src\xpath.jl:85
0-element Array{EzXML.Node,1}
Because the prefix was empty, it gets ignored. That seems to be why we get zero elements from findall
(maybe) 🤔
I think there's something about xml
being in the name of the tag...
julia> test5 = parsexml("""
<?xml version="1.0" encoding="utf-8"?>
<xbrl test="http">
</xbrl>""")
EzXML.Document(EzXML.Node(<DOCUMENT_NODE@0x00007fd996dfe450>))
julia> findall("/xbrl", test5)
1-element Array{EzXML.Node,1}:
EzXML.Node(<ELEMENT_NODE[xbrl]@0x00007fd996dcf980>)
julia> test6 = parsexml("""
<?xml version="1.0" encoding="utf-8"?>
<xbrl xmlns="http">
</xbrl>""")
EzXML.Document(EzXML.Node(<DOCUMENT_NODE@0x00007fd996fc2aa0>))
julia> findall("/xbrl", test6)
0-element Array{EzXML.Node,1}
And also, it seems to break parsing, every time I do that, all subsequent calls to parsexml
give me
ERROR: AssertionError: isempty(XML_GLOBAL_ERROR_STACK)
Stacktrace:
I think xmlns
is a special tag for namespacing, so those nodes get treated special somehow...
Oh, I see...
Your MWE is namespaced and findall(xpath, doc)
calls findall(xpath, doc.node, namespaces(doc.node))
, but
julia> namespaces(test.node)
0-element Array{Pair{String,String},1}
so no namespaces are being registered. I think that is why findall
is not working because there are no registered namespaces, but the root is namespaced (maybe) 🤔
From Wikipedia: https://en.wikipedia.org/wiki/XML_namespace
Namespace declaration
An XML namespace is declared using the reserved XML attribute xmlns or xmlns:prefix, the value of which must be a valid namespace name.
For example, the following declaration maps the "xhtml:" prefix to the XHTML namespace:
xmlns:xhtml="http://www.w3.org/1999/xhtml"
Any element or attribute whose name starts with the prefix "xhtml:" is considered to be in the XHTML namespace, if it or an ancestor has the above namespace declaration.
It is also possible to declare a default namespace. For example:
xmlns="http://www.w3.org/1999/xhtml"
In this case, any element without a namespace prefix is considered to be in the XHTML namespace, if it or an ancestor has the above default namespace declaration.
If there is no default namespace declaration in scope, the namespace name has no value.[6] In that case, an element without an explicit namespace prefix is considered not to be in any namespace.
Attributes are never subject to the default namespace. An attribute without an explicit namespace prefix is considered not to be in any namespace.
It seems like an issue dealing with default namespaces 🤔
EzXML apparently uses libxml2
and according to this:
http://xmlsoft.org/namespaces.html
default namespaces should be supported. I am probably confused 🤔
It works if I remove the default namespace:
julia> test = parsexml("""
<?xml version="1.0" encoding="utf-8"?>
<xbrl
xmlns:country="http://xbrl.sec.gov/country/2017-01-31"
xmlns:dei="http://xbrl.sec.gov/dei/2019-01-31"
xmlns:iso4217="http://www.xbrl.org/2003/iso4217"
xmlns:link="http://www.xbrl.org/2003/linkbase"
xmlns:mlic="http://www.metlife.com/20191231"
xmlns:srt="http://fasb.org/srt/2019-01-31"
xmlns:us-gaap="http://fasb.org/us-gaap/2019-01-31"
xmlns:xbrldi="http://xbrl.org/2006/xbrldi"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
</xbrl>""")
EzXML.Document(EzXML.Node(<DOCUMENT_NODE@0x0000000008349bf0>))
julia> findall("/xbrl", test)
1-element Array{EzXML.Node,1}:
EzXML.Node(<ELEMENT_NODE[xbrl]@0x000000003192c7e0>)
Ok. I am slowly learning about namespaces. We need a way to register default namespaces. This package is currently ignoring them 🤔
This is C#, but the discussion looks relevant: https://docs.microsoft.com/en-us/dotnet/standard/data/xml/xpath-queries-and-namespaces#the-default-namespace
The Default Namespace
In the XML document that follows, the default namespace with an empty prefix is used to declare the http://www.contoso.com/books namespace.
<books xmlns="http://www.contoso.com/books"> <book> <title>Title</title> <author>Author Name</author> <price>5.50</price> </book> </books>
XPath treats the empty prefix as the null namespace. In other words, only prefixes mapped to namespaces can be used in XPath queries. This means that if you want to query against a namespace in an XML document, even if it is the default namespace, you need to define a prefix for it.
For example, without defining a prefix for the XML document above, the XPath query /books/book would not return any results.
A prefix must be bound to prevent ambiguity when querying documents with some nodes not in a namespace, and some in a default namespace.
RTFM. Sorry for the noise 😔
Note: I'm on Windows 10, Julia v1.4.0, EzXML v1.1.0
Hi 👋
Thank you for this package 🙏
I am working on a fairly largish 45K line XML file (but some of the lines are VERY long) and can't seem to get a basic
findall
to work.I get my
It seems fine. Then I grab its root:
but then
I am expecting this to give me the root node.
Any idea what I'm doing wrong?
Edit:
This seems to work:
Edit^2: If it helps, here is the XML file:
https://www.sec.gov/Archives/edgar/data/937834/000093783420000005/mlic-12312019x10kdocum_htm.xml