Incorrect image from Facebook

[Thovi98]

Steps to reproduce

1. Setup Facebook url
2. Get image from Facebook

Expected behaviour

Have the image of the profil of the url

Actual behaviour

The new image is the the default avatar of Facebook (so not exactly an image). I have tried with different profiles, it’s always the same. And I tried with Instagram which works normaly.

When I try to get from Facebook for first time

When I try to update from Facebook an already updated Facebook image

Server configuration detail

Operating system: Linux 5.8.0-0.bpo.2-amd64 #1 SMP Debian 5.8.10-1~bpo10+1 (2020-09-26) x86_64

Webserver: nginx/1.14.2 (fpm-fcgi)

Database: mysql 10.3.25

PHP version:

7.3.23-4+0~20201018.71+debian10~1.gbpfc8934 Modules loaded: Core, date, libxml, openssl, pcre, zlib, filter, hash, Reflection, SPL, sodium, session, standard, cgi-fcgi, mysqlnd, PDO, xml, apcu, bcmath, bz2, calendar, ctype, curl, dom, mbstring, fileinfo, ftp, gd, gettext, gmp, iconv, igbinary, imagick, imap, intl, json, ldap, exif, mysqli, pdlib, pdo_mysql, apc, posix, readline, redis, shmop, SimpleXML, smbclient, sockets, sysvmsg, sysvsem, sysvshm, tokenizer, wddx, xmlreader, xmlwriter, xsl, zip, Phar, libsmbclient, Zend OPcache

Nextcloud version: 20.0.1 - 20.0.1.1

Updated from an older Nextcloud/ownCloud or fresh install: Updated from 19.04

Where did you install Nextcloud from: Yunohost (https://github.com/yunohost-apps/nextcloud_ynh)

Signing status

Integrity checker has been disabled. Integrity cannot be verified.

List of activated apps

``` Enabled: - accessibility: 1.6.0 - audioplayer: 2.12.0 - calendar: 2.1.2 - cloud_federation_api: 1.3.0 - comments: 1.10.0 - contacts: 3.4.1 - contactsinteraction: 1.1.0 - cospend: 1.1.4 - dashboard: 7.0.0 - dav: 1.16.0 - deck: 1.1.2 - documentserver_community: 0.1.8 - facerecognition: 0.7.0 - federatedfilesharing: 1.10.1 - federation: 1.10.1 - files: 1.15.0 - files_external: 1.11.1 - files_pdfviewer: 2.0.1 - files_rightclick: 0.17.0 - files_sharing: 1.12.0 - files_trashbin: 1.10.1 - files_versions: 1.13.0 - files_videoplayer: 1.9.0 - firstrunwizard: 2.9.0 - forms: 2.0.4 - integration_discourse: 0.0.5 - integration_google: 0.0.9 - issuetemplate: 0.7.0 - logreader: 2.5.0 - lookup_server_connector: 1.8.0 - mail: 1.5.1 - maps: 0.1.8 - metadata: 0.12.0 - news: 15.0.6 - nextcloud_announcements: 1.9.0 - notes: 4.0.0 - notifications: 2.8.0 - oauth2: 1.8.0 - onlyoffice: 6.1.0 - password_policy: 1.10.1 - phonetrack: 0.6.5 - photos: 1.2.0 - polls: 1.5.7 - privacy: 1.4.0 - provisioning_api: 1.10.0 - recommendations: 0.8.0 - riotchat: 0.6.13 - serverinfo: 1.10.0 - settings: 1.2.0 - sharebymail: 1.10.0 - spreed: 10.0.1 - support: 1.3.0 - survey_client: 1.8.0 - systemtags: 1.10.0 - tasks: 0.13.5 - text: 3.1.0 - theming: 1.11.0 - timemanager: 0.1.4 - twofactor_backupcodes: 1.9.0 - user_ldap: 1.10.2 - user_status: 1.0.0 - viewer: 1.4.0 - weather_status: 1.0.0 - workflowengine: 2.2.0 Disabled: - activity - admin_audit - cookbook - encryption - quicknotes - updatenotification - weather ```

Configuration (config/config.php)

``` { "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "localhost", "nextcloud.next.nohost.me" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "20.0.1.1", "overwrite.cli.url": "https:\/\/nextcloud.next.nohost.me", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "mysql.utf8mb4": true, "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "instanceid": "***REMOVED SENSITIVE VALUE***", "ldapIgnoreNamingRules": false, "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory", "updatechecker": false, "memcache.local": "\\OC\\Memcache\\APCu", "integrity.check.disabled": true, "filelocking.enabled": true, "memcache.locking": "\\OC\\Memcache\\Redis", "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": "6379", "timeout": "0.0", "password": "***REMOVED SENSITIVE VALUE***" }, "hashing_default_password": true, "logout_url": "https:\/\/next.nohost.me\/yunohost\/sso\/?action=logout", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_smtpmode": "smtp", "mail_sendmailmode": "smtp", "mail_domain": "***REMOVED SENSITIVE VALUE***", "mail_smtpauthtype": "LOGIN", "mail_smtpauth": 1, "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpport": "587", "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***", "mail_smtpsecure": "tls", "maintenance": false, "loglevel": 2 } ```

Are you using external storage, if yes which one: Local

Are you using encryption:

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...

LDAP configuration (delete this par if not used)

``` _lastChange: 1603795276background_sync_interval: 1800background_sync_offset: 0background_sync_prefix: enabled: yesinstalled_version: 1.10.2ldap_base: dc=yunohost,dc=orgldap_base_groups: ou=groups,dc=yunohost,dc=orgldap_base_users: ou=users,dc=yunohost,dc=orgldap_cache_ttl: 600ldap_configuration_active: 1ldap_display_name: displaynameldap_email_attr: mailldap_expert_username_attr: uidldap_group_display_name: cnldap_group_filter: objectClass=posixGroupldap_group_filter_mode: 0ldap_groupfilter_objectclass: posixGroupldap_host: localhostldap_login_filter: (&(|(objectclass=posixAccount))(uid=%uid))ldap_login_filter_mode: 0ldap_port: 389ldap_quota_attr: userquotaldap_tls: 0ldap_user_display_name: cnldap_user_filter_mode: 0ldap_userfilter_objectclass: posixAccountldap_userlist_filter: objectclass=posixAccounts01_lastChange: 1601502182s01ldap_configuration_active: 0types: authentication ```

Client configuration

Browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:82.0) Gecko/20100101 Firefox/82.0

Operating system:

Logs

Web server error log

``` Insert your web server log here ```

Nextcloud log

``` Insert your Nextcloud log here ```

Browser log

Insert your browser log here, this could for example include: a) The javascript console log b) The network log c) ...

[skjnldsv]

cc @call-me-matt

[call-me-matt]

oh no. you are right, the facebook API seems not to work anymore:-/ What a pity! Seems they enforce authentication now, which I don't know how we could implement for Nextcloud.

Requirements Change This endpoint supports App-Scoped User IDs (ASID), User IDs (UID), and Page-Scoped User IDs (PSID). Currently you can query ASIDs and UIDs with no requirements. However, beginning October 24, 2020, an access token will be required for all UID-based queries.

Lets keep our fingers crossed they dont do the same with Insta. @skjnldsv I'm afraid we need to deactivate facebook from sync until we have a new idea.

[Nickno95]

Have users generate an access token and store it so it can be used in API calls.

[call-me-matt]

Have users generate an access token and store it so it can be used in API calls.

So we need a facebook integration app, I suppose?

If I get it right, it would mean that administrators need to have a facebook account, create an application on facebook, connect it to Nextcloud and then also users need a facebook account which they grant access on Nextcloud. Already quite complex. And I did not completely understand if this app then can read profile pictures of their friends - or if they all have to allow the created app to read their profile. Anyone with experience on that?

[Thovi98]

Maybe we can ask to @eneiluj which I think developped the integration apps.

And I don’t know a lot, but on Android I can use CoSy for Facebook, which can sync profile pictures after I haved logged in on Facebook.

[skjnldsv]

@call-me-matt, what we can do is fetch the raw page of the profile (https://facebook.com/Nextclouders), and fetch the meta image in the source <meta property="og:image"..., it contains the profile picture with the size 200x200.

[call-me-matt]

what we can do is fetch the raw page of the profile

I think this works only for public profiles and pages. Other profiles do work also, but only very limited and low-res. After a couple of requests, the login-page is shown instead.

on Android I can use CoSy for Facebook, which can sync profile pictures

I noticed they also have a delay of several seconds between each request.

[Thovi98]

I noticed they also have a delay of several seconds between each request.

Yes, but only for importing friends, not for synchronisation, as it appears : https://mflisar.github.io/cosy/why-is-import-slow.html

[call-me-matt]

Still, there seems to be a pretty low limit. Accessing profile pages without being logged in only works a few times until you keep being redirected to the login page.

Try it out by replacing your file:

apps/contacts/lib/Service/Social/FacebookProvider.php

```php httpClient = $httpClient->NewClient(); } /** * Returns the profile-id * * @param {string} the value from the contact's x-socialprofile * * @return string */ public function cleanupId(string $candidate):string { try { if (strpos($candidate, 'http') !== 0) { $candidate = 'https://www.facebook.com/' . $candidate; } } catch (Exception $e) { $candidate = null; } return $candidate; } /** * Returns the profile-picture url * * @param {string} profileId the profile-id * * @return string|null */ public function getImageUrl(string $profileUrl):?string { try { $result = $this->httpClient->get($profileUrl); $htmlResult = $result->getBody(); $avatar = '/profilePicThumb">[ \n\t]*]+src="(http[^"]+)/'; if (preg_match($avatar, $htmlResult, $matches)) { return $matches[1]; } // keyword not found, maybe page changed? return null; } catch (Exception $e) { return null; } } } ```

[Thovi98]

Try it out by replacing your file: apps/contacts/lib/Service/Social/FacebookProvider.php

I tried but it cannot fetch any tried profile picture... But the default Facebook picture doesn’t update.

[eleith]

i haven't dug into the logs, but i'm also seeing a http 400 on any attempt to download an avatar from instagram

[call-me-matt]

It seems facebook profiles are accessible again without login. Not per API, but still. I just cannot figure out how to modify my request/regex, maybe someone can help me with that? see here: https://github.com/nextcloud/contacts/pull/1920#discussion_r623952910

[schklom]

It seems facebook profiles are accessible again without login. Not per API, but still. I just cannot figure out how to modify my request/regex, maybe someone can help me with that? see here: #1920 (comment)

Can you write what your regex looks like, and give a few strings it should match ? I'm not a web-developer, so I have no idea what you're trying to do in that PR. And although I'm not a pro, I know a bit how to use regex. Even if it's too high level for me, maybe someone more knowledgeable can help :)

[call-me-matt]

Hi schklom. Thank you for offering help! This was my attempt to fetch the image from facebook:

$result = $this->httpClient->get($profileUrl);
$htmlResult = $result->getBody();

$avatar = '/meta property="og:image" content="(http[^"]+)/';

if (preg_match($avatar, $htmlResult, $matches)) {
    return html_entity_decode($matches[1]);
}

But the page has changed and the profile picture is not tagged with og:image anymore. Take a look at the facebook page's source code (most comfortable with the "inspector" tool from your browser). And besides that, the page seems to load it's content in my browser, but not through the php request posted above. So I do not know how to grab the whole page in first place.

[schklom]

On Nextclouders.de's page + 2 friends' pages, I found that searching (on the Inspector tool) for 200x200 (sometimes p200x200, sometimes s200x200) returns 2-3 results, but only 1 that looks like either: <image ....... xlink:href="https://scontent-arn2-1.xx.fbcdn.net/v/t......./p200x200/........"></image> or <img ........ src="https://scontent-arn2-2.xx.fbcdn.net/v/t....../p200x200/........" ......>

It seems these urls are only valid for ~30s, and I have no clue if they appear with PHP. The one that exists contains the profile picture when there is one.

If they do, I think this regex should work <image.*?xlink\:href="(https\://.*?/[a-zA-Z]?200x200/[^"]*?)"></image>) for the 1st case.

For the 2nd, try <img.*?src="(https\://[^"]*?)/[a-zA-Z]?200x200/[^"]*?".*?>.

Maybe remove the ? in [a-zA-Z]?, I only tried 3 URLs.

Hopefully it helps :)

[call-me-matt]

Thank you for your work. Now we need to figure out how to load the page with php, as the request from my code seems not to load the complete page.

[schklom]

the request from my code seems not to load the complete page

This is strange. A basic curl https://www.facebook.com/Nextclouders.de/ on my Windows CLI gives me a link with the 200x200 string. It takes me too long to verify that it works since the link is only valid ~30s, maybe you will have more luck. After a few curls, I get redirected to login screen, maybe this is what's happening to you?

After searching a little it seems others had success simulating a browser via curl in PHP and setting the appropriate headers. I think the lack of a User-Agent makes it obvious that the request doesn't come from a human being, and maybe they check other headers too. Maybe the function from httpClient you use can do that? I have no clue.

Here is the link of a guy who had some success if it helps https://stackoverflow.com/a/55829622

In order to check which headers to send, you can check the ones your browser sends when you connect to facebook. Visit https://www.facebook.com/Nextclouders.de, open Inspector, then go to Network, reload the page, click the line with Status: 200, Method: GET, Domain: www.facebook.com, check its Request Headers, then try to put them in your PHP request. If it still doesn't work, then I'm lost 🤷

[call-me-matt]

Well, it works for public pages (like nextclouders) but it does not work for profiles (try it out for https://www.facebook.com/zuck). It seems like a basic page is loaded and only after a moment the actual content is reloaded. Any idea how to fetch that with php?

[jivanpal]

It seems like a basic page is loaded and only after a moment the actual content is reloaded. Any idea how to fetch that with php?

@call-me-matt This definitely requires executing some JavaScript that is loaded on the actual profile page. Programs like HtmlUnit exist that can facilitate this, but HtmlUnit in particular would require a Java runtime on the server, which PHP can then hook into. I don't know of any PHP library or anything that allows this to be done all in PHP instead.

Probably best just to have the Nextcloud user log in / connect to Facebook, then we have a Facebook Client Access Token that can be used to query the Facebook Graph API and get the image.

[XCoolBeatX]

Hi @call-me-matt,

Following on this thread. Was there any developments on this matter eventually.

To be honest, I get the feeling that this profile image updating concept kinda died, based on the fact that there was no other threads on this and Google+ is still listed as a social network.

[call-me-matt]

Hi. No further development for pictures from facebook. But this profile image updating concept is not dead! Here, you can see all the providers that are implemented and active:

• Mastodon
• Tumblr
• Diaspora
• Xing
• Telegram
• Gravatar

[XCoolBeatX]

@call-me-matt Aaaaa thank you for the clarification. Honestly, most people have Facebook. As the saying goes "everyone and their dog is on Facebook". So I really would like to see renewed development in a Facebook Integration plugin.

Unfortunately I can't give much development help (as I lack the skillset) but I would be more then happy to do any testing and QAs