Open ururk opened 6 months ago
How were you calling unstructured
when you saw this problem?
When partitioning you can set the content_type
parameter to specify the content type when you know it and the auto-recognition has trouble with it.
I'm using the UnstructuredURLLoader:
loader = UnstructuredURLLoader(urls=["https://sites.google.com/umich.edu/mm-post-award-manual/project-management/cost-sharing-commitments-internal-agreements?authuser=0"], continue_on_failure=False, headers={'User-Agent': 'Mozilla/5.0'})
Internally this calls (pseudo-code):
from unstructured.partition.auto import partition
elements = partition(
url=url, headers=self.headers, **self.unstructured_kwargs
)
I suppose this "bug report" might be better put in the UnstructuredURLLoader
library (since partition
takes a content_type
param), but the bigger problem is, the document starts with a <!DOCTYPE html>
declaration, which could factor into determining whether it was a csv or not.
The other thing I should point out, charset
is a valid directive in the Content-Type header, and isn't non-standard:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type
FYI - the comma might be removed from the page title, which may fix the issue - see Additional Context for what text it was using to detect CSV
Yeah, I was just typing that :) This is a case we should handle.
You can try the content_type
argument as a workaround for now, but I'll see about getting this fixed up.
I believe you can add {"content_type": "text/html"}
as the unstructured_kwargs
argument to the UnstructuredURLLoader
call to pass it along to unstructured
.
Yeah, I was just typing that :) This is a case we should handle.
Thanks!
You can try the
content_type
argument as a workaround for now, but I'll see about getting this fixed up.I believe you can add
{"content_type": "text/html"}
as theunstructured_kwargs
argument to theUnstructuredURLLoader
call to pass it along tounstructured
.
Ah - yes I believe that would work. Since this is part of a bigger, automated project I'm not able to do a one-off tweak like that (but something to consider).
Describe the bug I came across a webpage which is being detected as a CSV file. It should be detected as html. The page, unfortunately, returns its content type as:
Content-Type: text/html; charset=utf-8
To Reproduce
Expected behavior I expect it to detect the page as html, it detects it as CSV.
Additional context When the page is read in, only a portion of it is read and because Google Sites formats HTML a certain way, lines get split up like: