Feature Request: Make {lang} and {idnumber} available in other tags

[jaschrock]

I like the {scrape} tag and it seems to work for bringing in some raw content. A couple of things I don't seem to be able to get to work:

1. I would like to use the other filter tags to create some URL variables (for instance {lang}. It gives an xml parsing error when I try this. though, so I am guessing it doesn't like the nested {} within the scrape tag. I tried (though wasn't hopeful) to urlencode the {} but that didn't work, naturally.
2. I am wanting a shorter timeout period for this one location where i am placing the code. I would rather it not scrape if the site isn't there than hang on for maybe 10 seconds or so. Is there a way to set the timeout value for the scrape to fail or can it be added as an option to the scrape tag? I wasn't sure if this was a moodle, php default or something else.

Thanks for your work on these tools, they are very helpful!

[michael-milette]

Hi @jaschrock

I don't think it has anything to do with nesting FilterCode tags. If the information was available, Moodle, as an external website, would have likely already processed the tags.

Moodle cannot scrape from Moodle as you might assume. When you use the {scrape} tag from a Moodle page, it is your web server that will be accessing your Moodle site, not you as a user coming from a web browser. Since the server is not logged in as you, with few exceptions like the login page and other publicly accessible pages, it will be unable to access the content that you are most likely are trying to access.

As for the timeout, I think the code uses PHP's DOMDocument::loadHTML (would need to double check). Unfortunately there are no timeout options for it.

You might be able to change this by modifying your php.ini file. I have not tested it but I think the default_socket_timeout setting affects the scrape tag. HOWEVER, changing this will also affect other operations which could potentially reduce the stability of your whole site.

I don't have time to re-write the code to add support for a custom timeout right now. I might have some time to work on it during July or August. However, I am open to offering my services to you which would get it done much quicker. Let me know if you are in a hurry for it.

Of course I am always open to pull requests if you think that you have the skills to re-write the {scrape} part of the code.

Best regards,

Michael

[jaschrock]

Hi Michael, Thanks for your response. I figured the timeout might be more complicated. It might be a nice feature to add into a bucket at some point, but no rush. I am not trying to scrape from Moodle. What I am trying to do is to pull from another site and send URL variables to that site. Something like: {scrape url="https://mysite.example.com/mypage.jsp?id={idnumber}" tag="body"} I put this in a label. So, I am trying to pass, using your filter code {idnumber} and passing that (along with language and an external identifier) to this other site which will then use it to pull the correct information on a particular student and pass back some data (in this case, open evaluations they should do). However, when I save this, I get: "Exception - String could not be parsed as XML" so, that is why I was figuring it was not happy with the nested tag. Does that make sense? Is there a way I can put the src in a variable and build it and pass it to the scrape command? Thanks Alan

[michael-milette]

Hi Alan,

Ah! Now I understand. That indeed would be a problem. Not all FilterCode tags will work inside other FilterCode tags. In your case, the beginning of the tag is seen as "{scrape..." and ends with "... idnumber} because that is the first closing "}" it encouters. However, I might be able to help you out with that.

May I assuming that {idnumber} is the actual tag that you actually want to use inside the {scrape} tag, or was this just an example? I cannot fix the problem for all tags but I think that I can shuffle some code around a little to possibly have {idnumber} converted before the {scrape} tag.

Please confirm that you actually want to use {idnumber} (the user's idnumber from their profile) before I begin any work on this issue.

Best regards,

Michael

[jaschrock]

Hi Michael,

There are actually two tags I would need - {idnumber} and {lang}. That would be excellent!

Thanks

Alan

[michael-milette]

Hi @jaschrock ,

{idnumber} and {lang} should now both be available for use inside the {scrape} tag as well as other tags in the version of FilterCodes here on GitHub.

If you would be so kind, please try out the latest version and let me know if it works for you.

If it does, this change will be included in the upcoming release of FilterCodes on Moodle.org/plugins around the time of the release of Moodle 4.0.

Best regards,

Michael

[jaschrock]

Hi Michael,

That is great!

I just tested it and all seems good!

Thanks!

Alan

[michael-milette]

Hi Alan,

Thank you for testing it and getting back to me.

Best regards,

Michael