Xibo to support RSS feeds from Microsoft Sharepoint

[Mondlicht81]

Hi,

I'm using Xibo in connection with a Microsoft Sharepoint environment. I had some difficulties to get the RSS feeds working, but I finally managed to get them displayed.

The problem is that Sharepoint uses Kerberos and NTLM for authentication. This isn't enabled by SimplePie. So I modified xibo/3rdparty/simplepie/library/SimplePie/File.php by adding some cURL options for NTLM and GSS negotiation. See also here: http://stackoverflow.com/questions/15697157/using-curl-with-ntlm-auth-to-make-a-post-is-failing

I suggest to do following changes:

diff -r xibo/3rdparty/simplepie/library/SimplePie/File.php xibo-cms-1.7.4/3rdparty/simplepie/library/SimplePie/File.php
67,77d66
<   
<   private function removeDoubledHeader($input)
<   {
<       
<       if (substr_count($input,'HTTP/1.1') > 1)
<       {
<           $input = substr($input,strrpos($input,'HTTP/1.1'));
<       }
< 
<       return $input;
<   }
121,122d109
<               curl_setopt($fp, CURLOPT_HTTPAUTH, CURLAUTH_NTLM | CURLAUTH_GSSNEGOTIATE);
<               curl_setopt($fp, CURLOPT_FOLLOWLOCATION, TRUE);
139d125
< 
161,163c147
<                   $this->headers = $this->removeDoubledHeader($this->headers);
<                   
<                   Debug::Audit('Headers: ' . var_export($this->headers, true));

---
>                   //Debug::Audit('Headers: ' . var_export($this->headers, true));
170d153
<                       Debug::Audit('Body: ' . var_export($this->body, true));
172d154
<                       Debug::Audit('Status Code: ' . var_export($this->status_code, true));                       
diff -r xibo/3rdparty/simplepie/library/SimplePie.php xibo-cms-1.7.4/3rdparty/simplepie/library/SimplePie.php
1379c1379
<           $this->error = sprintf('This XML document is invalid, likely due to invalid characters. XML error: %s at line %d, column %d, content: %s', $parser->get_error_string(), $parser->get_current_line(), $parser->get_current_column(), $this->raw_data);

---
>           $this->error = sprintf('This XML document is invalid, likely due to invalid characters. XML error: %s at line %d, column %d', $parser->get_error_string(), $parser->get_current_line(), $parser->get_current_column());

RSS feeds without authentication are not affected at all. With authentication , e.g. https://user:pass@mywebsite.net will use NTLM and GSS negotiation. There's another option CURLAUTH_ANYSAFE which I didn't try yet, but might be more general.

I also added some more debug logs, which are not necessary but helpful in debugging.

Do you think this could be taken over into the official code?

[dasgarner]

I don't see why we couldn't adopt this in the official code, although i'll need to look in more detail. I've tagged it up accordingly.

Thanks a lot for contributing!

[dasgarner]

1.8 has moved feed parsing from SimplePie which was unmaintained to Picofeed and therefore I don't feel comfortable including this patch until we also have a patch for 1.8 which does the same.

I've moved this to backlog for the moment until the above can be done.

[Mondlicht81]

Hi,

I set up a xibo cms version 1.8.0 inside docker now. (Cost some time to get it running...) I'm currently trying to set the same options inside Picofeed, but somehow it still doesn't like to get the feeds. It ends up with maximum of redirections reached. I'll inform you when I succeeded.

But by the way, I was struggling with the proxy exceptions, also in 1.7.x already.

If you do following change in /var/www/cms/lib/Service/ConfigService.php, then the proxy exceptions are correctly checked.

        /**
        * Should the host be considered a proxy exception
        * @param $host
        * @return bool
        */
        public function isProxyException($host)
        {

        $proxyException = $this->GetSetting('PROXY_EXCEPTIONS');
        /* return ($proxyException != '' && stripos($host, $proxyException) > -1); */
        $strippedHost=parse_url($host,1);
        $proxyExceptionArray = explode(',', $proxyException);
        foreach ($proxyExceptionArray as $proxyExceptionArrayEntry){
                if (stripos($strippedHost, $proxyExceptionArrayEntry) !== false)
                {
                        /* Debug::Audit($strippedHost . ' in ' . $proxyException . '. Included in: ' . $proxyExceptionArrayEntry); */
                        return (true);
                }

        }

         return (false);
       }

[dasgarner]

OK, looks interesting - could you please create a separate issue for Proxy exceptions and explain why you have a problem with the current code?

We're currently matching against full URLs - are you suggesting a change to matching hosts only?

Can you put that detail in the new issue?

Thanks!

[Mondlicht81]

Hi Dan, Yes, exactly, with this change it's matching the host only, as e.g. in Firefox. Ok, I'll open a new issue for that. https://github.com/xibosignage/xibo/issues/1096

[Mondlicht81]

Finally, I made it work. A CURLOPT_VERBOSE which prints into /var/log/apache2/error.log saved me. Then I could compare it to curl https://mysharepoint.net --ntlm --negotiate -v

Following changes are needed in /var/www/cms/vendor/fguillot/picofeed/lib/PicoFeed/Client/Curl.php: Add curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANYSAFE); and modify curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);

/**
     * Prepare curl context.
     *
     * @return resource
     */
    private function prepareContext()
    {
        $ch = curl_init();

        curl_setopt($ch, CURLOPT_URL, $this->url);
        curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
        curl_setopt($ch, CURLOPT_TIMEOUT, $this->timeout);
        curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $this->timeout);
        curl_setopt($ch, CURLOPT_USERAGENT, $this->user_agent);
    curl_setopt($ch, CURLOPT_HTTPHEADER, $this->prepareHeaders());
        curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANYSAFE);
        curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
        curl_setopt($ch, CURLOPT_ENCODING, '');
        curl_setopt($ch, CURLOPT_COOKIEJAR, 'php://memory');
        curl_setopt($ch, CURLOPT_COOKIEFILE, 'php://memory');

        // Disable SSLv3 by enforcing TLSv1.x for curl >= 7.34.0 and < 7.39.0.
        // Versions prior to 7.34 and at least when compiled against openssl
        // interpret this parameter as "limit to TLSv1.0" which fails for sites
        // which enforce TLS 1.1+.
        // Starting with curl 7.39.0 SSLv3 is disabled by default.
        $version = curl_version();
        if ($version['version_number'] >= 467456 && $version['version_number'] < 468736) {
            curl_setopt($ch, CURLOPT_SSLVERSION, 1);
        }

        $ch = $this->prepareDownloadMode($ch);
        $ch = $this->prepareProxyContext($ch);
        $ch = $this->prepareAuthContext($ch);

        return $ch;
    }

Additionally, remove the Connection:close from the HTTP header (I just replaced it with ' '):

/**
     * Prepare HTTP headers.
     *
     * @return string[]
     */
    private function prepareHeaders()
    {
        $headers = array(
           ' ', 
       //'Connection: close',
        );

        if ($this->etag) {
            $headers[] = 'If-None-Match: '.$this->etag;
            $headers[] = 'A-IM: feed';
        }

        if ($this->last_modified) {
            $headers[] = 'If-Modified-Since: '.$this->last_modified;
        }

        $headers = array_merge($headers, $this->request_headers);

        return $headers;
    }

The two Curl options were similar to my changes in SimplePie. But the Connection: close was not so easy to find. Maybe this should be discussed with Frédéric and the other PicoFeed authors to learn more about the reason behind.

[Mondlicht81]

Hi, What's your plan to take this change over?

[dasgarner]

Thank you for the follow up - we will need to wait for the issue in PicoFeed to be resolved before it can be included in Xibo. The only alternative being to fork that project, which I don't want.

Were you prepared to submit a PR for that project to review? I didn't mention it because I saw you had already opened an issue with that project.

[Mondlicht81]

Unfortunately..... I got zero reaction to my ticket.

What exactly does PR mean in this context?

[dasgarner]

PR means "pull request", which is where you notice something wrong with some code, take a fork of the repository and fix it - you then submit a pull request, containing your fix, which the maintainer of the project can merge.

Due to the changes you've suggested, we do need to try and get it merged in the upstream project as there isn't any way to override those functions (they are private functions).

[Mondlicht81]

Thanks for the explanations, Dan. I finally found time to create the pull request, hope I did everything correctly.

https://github.com/miniflux/picoFeed/pull/342

[Mondlicht81]

Several months later, and still no reaction of the picoFeed guys. :-( I have no idea what to do.

[alexharrington]

Dan's not available right at the moment, but all I can suggest is trying to contact their devs and see if there's a reason they won't accept your PR. We don't have any control in that regard I'm afraid.

[Mondlicht81]

Very unfortunately, my ticket was closed at the picoFeed project. Reason: no longer maintained. Any suggestions?

[dasgarner]

We don't use PicoFeed to pull the data anymore - only to parse the result. It may be that what you want is now possible by modifying Xibo directly.

We now use Guzzle for this - see: https://github.com/xibosignage/xibo-cms/blob/develop/lib/Widget/Ticker.php#L831

[Mondlicht81]

Too bad. When I checked out the repository in November, it was still in. I didn't know about your plans to switch to another feed reader. Then I could have saved the effort. :-)

Anyway, the version of MS Sharepoint currently in use here doesn't support RSS feeds anymore. So maybe this whole topic can be closed now.

[dasgarner]

To be clear - we do still use the repository for feed parsing, just not feed fetching as it was unreliable.