getVocabulary: Call scrub_html on individual items

[reinhardt]

… but check for script/html first

Fixes JSONDecodeError when terms contain incomplete HTML

Unfortunately I did find a way to break #287, not in terms of security, but of invalid JSON. As the test demonstrates, terms with incomplete HTML result in broken JSON like this:

{"results": [{"id": "term 0 <b>", "text": "term 0 <b>"}, {"id": "term 1 <b>", "text": "term 1 <b>"}, {"id": "term 2 <b>", "text": "term 2 <b>"}], "total": 3}</b></b></b></b></b></b>

This PR reverts the approach of calling scrub_html on the end result, but adds a check for script or html on the individual items, which saves the time needed to invoke lxml if there's nothing to scrub out.

EDIT: The check was moved to https://github.com/plone/Products.PortalTransforms/pull/66

[mister-roboto]

@reinhardt thanks for creating this Pull Request and helping to improve Plone!

TL;DR: Finish pushing changes, pass all other checks, then paste a comment:

@jenkins-plone-org please run jobs

To ensure that these changes do not break other parts of Plone, the Plone test suite matrix needs to pass, but it takes 30-60 min. Other CI checks are usually much faster and the Plone Jenkins resources are limited, so when done pushing changes and all other checks pass either start all Jenkins PR jobs yourself, or simply add the comment above in this PR to start all the jobs automatically.

Happy hacking!

[reinhardt]

@jenkins-plone-org please run jobs

[reinhardt]

Thanks for the suggestion! I've removed the condition here, so this PR now mainly reverts #287, plus adds a regression test for valid json.

@jenkins-plone-org please run jobs

[reinhardt]

@jenkins-plone-org please run jobs

[mauritsvanrees]

I have released plone.app.content 4.1.6 and Products.PortalTransforms = 4.1.0.

I have updated https://dist.plone.org/release/6.0-dev/ and https://dist.plone.org/release/6.1-dev/