I'm going to propose that we set the default to False.
The current behavior (as of 0.4.4) is to load the entire text into the first element of the list (link), and I'd like to make that the default.
This would benefit us by making upgrading simpler. Our code that calls load_data expects to fetch the entire text using result[0].text. Upgrading would have surprised us by dropping pages. We'd have to spend time trackdown down the issue followed by adding seperate_page=False.
I cannot speak for other users, but we have documents whose parsed results contain \n---\n. This typically comes from a footer. Splitting on --- can "introduce" pages.
Summary
I'm going to propose that we set the default to
False.The current behavior (as of 0.4.4) is to load the entire text into the first element of the list (link), and I'd like to make that the default.
load_dataexpects to fetch the entire text usingresult[0].text. Upgrading would have surprised us by dropping pages. We'd have to spend time trackdown down the issue followed by addingseperate_page=False.\n---\n. This typically comes from a footer. Splitting on---can "introduce" pages.