Closed jjschwarz closed 2 months ago
Hey @jjschwarz
I am sorry that you have so much trouble with the plugin on different domains.
Can you check (with your hoster) if it might be a modsec issue? https://blog.rac.me.uk/2022/11/10/activitypub-for-wordpress-how-to-fix-modsecurity-to-make-it-work/
I also did some quick testing with curl against my user and the site user with the commands: curl "https://sydneyschwarz.com/@sydneyschwarz.com" -H 'Accept: application/activity+json' curl "https://sydneyschwarz.com/@jason" -H 'Accept: application/activity+json'
both seemed to return valid output
It is not about the GET
requests, you have to test a POST
requests to the inbox
Hey @jjschwarz
I am sorry that you have so much trouble with the plugin on different domains.
Can you check (with your hoster) if it might be a modsec issue? https://blog.rac.me.uk/2022/11/10/activitypub-for-wordpress-how-to-fix-modsecurity-to-make-it-work/
I am actually the hosting company and this is just another instance in Docker behind the exact same front end server. Given the other 3 on that server are running the plugin fine using the same Docker formula I am going to say it isn't the modsec issue.
It is not about the
GET
requests, you have to test aPOST
requests to the inbox
Yes, working on testing that next, just wanted to get the basics out of the way.
Unfortunately the POST is doing the same thing as the other two sites did previously:
Sep 04 08:24:31 mastodon.lothlorien.net bundle[876]: 2024-09-04T12:24:31.543Z pid=876 tid=2c3q50 WARN: {"context":"Job raised exception","job":{"retry":16,"queue":"push","dead":false,"args":["{\"@context\":\"https://www.w3.org/ns/activitystreams\",\"id\":\"https://mastodon.lothlorien.net/af189127-8fae-4e86-8c26-d8ddb0d1888a\",\"type\":\"Follow\",\"actor\":\"https://mastodon.lothlorien.net/users/jason\",\"object\":\"https://sydneyschwarz.com/author/jason/\"}",109307799703588249,"https://sydneyschwarz.com/wp-json/activitypub/1.0/actors/1/inbox",{"bypass_availability":true}],"class":"ActivityPub::DeliveryWorker","jid":"46f093ef00b85a3f10fd8fcf","created_at":1725452671.2590404,"enqueued_at":1725452671.2592108}} Sep 04 08:24:31 mastodon.lothlorien.net bundle[876]: 2024-09-04T12:24:31.543Z pid=876 tid=2c3q50 WARN: Mastodon::UnexpectedResponseError: https://sydneyschwarz.com/wp-json/activitypub/1.0/actors/1/inbox returned code 401
I get back a 401 error. Last time this was the index.php in the URL, however this is a fresh install, so in theory it wouldn't have that issue (and I do not see it).
Do you maybe have a chance to also debug the response message?
It seems that its an issue with the Signature Verification!
Trying to figure how to get Mastodon to let me capture the entire conversation.
Can you maybe also try to debug the value of $route
here: https://github.com/Automattic/wordpress-activitypub/blob/88675ea6c632367ee5d02358902fe7eba63d044a/includes/class-signature.php#L253
OK, so at line 253 I added: [Wed Sep 04 14:02:40.310632 2024] [php:notice] [pid 23:tid 23] [client 2620:128:2000:254:de3e:b475:eef6:27dd:0] Activitypub\Signature::verify_http_signature: route = /wp-json/activitypub/1.0/actors/1/inbox
and I see:
[Wed Sep 04 14:02:40.310632 2024] [php:notice] [pid 23:tid 23] [client 2620:128:2000:254:de3e:b475:eef6:27dd:0] Activitypub\Signature::verify_http_signature: route = /wp-json/activitypub/1.0/actors/1/inbox
OK, that seems to be fine.
Let's try the next one. Can you debug $actor
here: https://github.com/Automattic/wordpress-activitypub/blob/88675ea6c632367ee5d02358902fe7eba63d044a/includes/class-signature.php#L331 ???
Sorry, above I obviously meant to paste: error_log(METHOD . ": route = $route");
[Wed Sep 04 14:16:41.848501 2024] [php:notice] [pid 27:tid 27] [client 2620:128:2000:254::80:37638] Activitypub\Signature::get_remote_key: actor = Array\n(\n [@context] => Array\n (\n [0] => https://www.w3.org/ns/activitystreams\n [1] => https://w3id.org/security/v1\n [2] => Array\n (\n [manuallyApprovesFollowers] => as:manuallyApprovesFollowers\n [toot] => http://joinmastodon.org/ns#\n [featured] => Array\n (\n [@id] => toot:featured\n [@type] => @id\n )\n\n [featuredTags] => Array\n (\n [@id] => toot:featuredTags\n [@type] => @id\n )\n\n [alsoKnownAs] => Array\n (\n [@id] => as:alsoKnownAs\n [@type] => @id\n )\n\n [movedTo] => Array\n (\n [@id] => as:movedTo\n [@type] => @id\n )\n\n [schema] => http://schema.org#\n [PropertyValue] => schema:PropertyValue\n [value] => schema:value\n [discoverable] => toot:discoverable\n [Device] => toot:Device\n [Ed25519Signature] => toot:Ed25519Signature\n [Ed25519Key] => toot:Ed25519Key\n [Curve25519Key] => toot:Curve25519Key\n [EncryptedMessage] => toot:EncryptedMessage\n [publicKeyBase64] => toot:publicKeyBase64\n [deviceId] => toot:deviceId\n [claim] => Array\n (\n [@type] => @id\n [@id] => toot:claim\n )\n\n [fingerprintKey] => Array\n (\n [@type] => @id\n [@id] => toot:fingerprintKey\n )\n\n [identityKey] => Array\n (\n [@type] => @id\n [@id] => toot:identityKey\n )\n\n [devices] => Array\n (\n [@type] => @id\n [@id] => toot:devices\n )\n\n [messageFranking] => toot:messageFranking\n [messageType] => toot:messageType\n [cipherText] => toot:cipherText\n [suspended] => toot:suspended\n [memorial] => toot:memorial\n [indexable] => toot:indexable\n [focalPoint] => Array\n (\n [@container] => @list\n [@id] => toot:focalPoint\n )\n\n )\n\n )\n\n [id] => https://mastodon.lothlorien.net/users/jason\n [type] => Person\n [following] => https://mastodon.lothlorien.net/users/jason/following\n [followers] => https://mastodon.lothlorien.net/users/jason/followers\n [inbox] => https://mastodon.lothlorien.net/users/jason/inbox\n [outbox] => https://mastodon.lothlorien.net/users/jason/outbox\n [featured] => https://mastodon.lothlorien.net/users/jason/collections/featured\n [featuredTags] => https://mastodon.lothlorien.net/users/jason/collections/tags\n [preferredUsername] => jason\n [name] => Jason Schwarz\n [summary] =>
Chief Technical Officer of MSC Inc.
N4JJS on the Air and Twitter
GPG Key 0xFC40CF386FE81B07
Fingerprint 9BF7\xc2\xa00695\xc2\xa06403\xc2\xa0F360\xc2\xa09DBD\xc2\xa0\xc2\xa051D8\xc2\xa0FC40\xc2\xa0CF38\xc2\xa06FE8\xc2\xa01B07
I just grabbed the first value from the array, but that seems to point back to my jason@lothlorien.net account, which is the same one I am using with all of the working servers.
Also fine!
Now can you check $verified
here: https://github.com/Automattic/wordpress-activitypub/blob/88675ea6c632367ee5d02358902fe7eba63d044a/includes/class-signature.php#L315
OK, this is odd, I just get back a blank value for $verified on that line:
[Wed Sep 04 14:25:16.827821 2024] [php:notice] [pid 26:tid 26] [client 2620:128:2000:254:de3e:b475:eef6:27dd:0] Activitypub\Signature::verify_http_signature: verified =
OK, then the openssl_verify
is the issue! can you debug every param: $signed_data
, $signature_block['signature']
, $public_key
, $algorithm
?
Was already working on that :)
[Wed Sep 04 14:33:12.478907 2024] [php:notice] [pid 76:tid 76] [client 2620:128:2000:254:de3e:b475:eef6:27dd:0] Activitypub\Signature::verify_http_signature: signed_data = (request-target): post /wp-json/activitypub/1.0/actors/1/inbox\nhost: sydneyschwarz.breepriv.lothlorien.net\ndate: Wed, 04 Sep 2024 14:33:12 GMT\ndigest: SHA-256=a64iQuJfFNWG/bzlRhU3jymEfeYCeZWFwK20wkEkoGg=\ncontent-type: application/activity+json [Wed Sep 04 14:33:12.478941 2024] [php:notice] [pid 76:tid 76] [client 2620:128:2000:254:de3e:b475:eef6:27dd:0] Activitypub\Signature::verify_http_signature: signature_block = 1 [Wed Sep 04 14:33:12.478953 2024] [php:notice] [pid 76:tid 76] [client 2620:128:2000:254:de3e:b475:eef6:27dd:0] Activitypub\Signature::verify_http_signature: public_key = -----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAyb9cpS8K5Yos9ZUqDUra\nVdnxm5ybItRMA7AhMCnfqLUyTtOvEjjT6w2wi4jeX1WFe0TZtkzSCN2jIpSQjS1u\nCnK/HK7kjyBkZDiPDZ2HMOskp8efF4YRtP343gZ/4qoE3ekMQyAG5j+GYYvqIwrB\nvNrK3KGfUOVe99XK6PrOhPEMLVWqHw/X1qVPu9QCklmdbrk6N7/WCLyZvP6KttEt\nEsNK06rfD1rR9ZsvI9tCOnUjnfRoW9/46V8HfUhy9iA/ubzf6SOgHcTlLZpXZghM\ntuw0oFvl/URmhNzURtgRU+V4jH8eCqjZXJ5TmTp5eWuzURtrDBId6tRq9vwLVWj1\nFQIDAQAB\n-----END PUBLIC KEY----- [Wed Sep 04 14:33:12.478965 2024] [php:notice] [pid 76:tid 76] [client 2620:128:2000:254:de3e:b475:eef6:27dd:0] Activitypub\Signature::verify_http_signature: algorithm = sha256
$signature_block
is 1
? 😳
Sorry, needed to go deeper on signature_block:
[Wed Sep 04 14:35:11.365394 2024] [php:notice] [pid 35:tid 35] [client 2620:128:2000:254::80:36386] Activitypub\Signature::verify_http_signature: signed_data = (request-target): post /wp-json/activitypub/1.0/actors/1/inbox\nhost: sydneyschwarz.breepriv.lothlorien.net\ndate: Wed, 04 Sep 2024 14:35:11 GMT\ndigest: SHA-256=vSwAVVOmSN6j2MIEb3+6zFI6MT7qxNGkDSabvcSr7SA=\ncontent-type: application/activity+json [Wed Sep 04 14:35:11.365416 2024] [php:notice] [pid 35:tid 35] [client 2620:128:2000:254::80:36386] Activitypub\Signature::verify_http_signature: signature_block = 0\x8e\x96i\xba(9]5cY\xd7\x15\xef\x0fK\x95/\xf4}5\x16\xcb\x93\X\xc9\xe8\x9ea\xd80\b\x9b\xe2\x05#\xa5\xe0h\x99\x84\x9fYrB\xf2=\x82D/Z)\x86\x06\x16B\xe4v\x98M\xb3O\x994\xd2*,}\xceW]\x8f\xbbQ\r$\xcd\xaae\xc5\xd2\x15\xe1&\x92.\xe9\xad\xd5\xd4[\xbf4\x11\x16i\xa1\xc11\xd2(\r\xaf\xed\xc2\xe7\xffY\xdd\xbf=\xed\xdf\xad\xf9\xc1\xa4(\xb4\xc0\xb1\x9f\xc9f!!P\xed\xed\xf6\x07\xd1\xf1OC\x9f\xe7\xd7\x14\xc1\t4S0Pc"~\x03\xd9\xe2\x8fpc\x1f\xf7\xbe.\x8c\x8a\xf6\x93"\x9b\x8c\xe5A\xfd\xdb\x87\x93\x89w\x8f\xd6ab\x99\xc0\xdf\x07\x07\xbf\x92=\xdb\xfa\xed[&\x8e\x82=\xbb\xaa\xf0]\x03\x88\xa1\xb6|g\xa0t\x14\x88\x13j\xeda\xe9cG\xb7d\xf3\xaa\xf6\x07\xf5+\x95\xc5\x113\x1du\xe6\x9bp\x02\x01\x8bq\xf7\xac\xdc+9\xeb\xc0b\x03\x906]\xd2\xf3\x89\x14\x8cQ\x80\x19 [Wed Sep 04 14:35:11.365433 2024] [php:notice] [pid 35:tid 35] [client 2620:128:2000:254::80:36386] Activitypub\Signature::verify_http_signature: public_key = -----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAyb9cpS8K5Yos9ZUqDUra\nVdnxm5ybItRMA7AhMCnfqLUyTtOvEjjT6w2wi4jeX1WFe0TZtkzSCN2jIpSQjS1u\nCnK/HK7kjyBkZDiPDZ2HMOskp8efF4YRtP343gZ/4qoE3ekMQyAG5j+GYYvqIwrB\nvNrK3KGfUOVe99XK6PrOhPEMLVWqHw/X1qVPu9QCklmdbrk6N7/WCLyZvP6KttEt\nEsNK06rfD1rR9ZsvI9tCOnUjnfRoW9/46V8HfUhy9iA/ubzf6SOgHcTlLZpXZghM\ntuw0oFvl/URmhNzURtgRU+V4jH8eCqjZXJ5TmTp5eWuzURtrDBId6tRq9vwLVWj1\nFQIDAQAB\n-----END PUBLIC KEY----- [Wed Sep 04 14:35:11.365448 2024] [php:notice] [pid 35:tid 35] [client 2620:128:2000:254::80:36386] Activitypub\Signature::verify_http_signature: algorithm = sha256 [Wed Sep 04 14:35:11.365460 2024] [php:notice] [pid 35:tid 35] [client 2620:128:2000:254::80:36386] Activitypub\Signature::verify_http_signature: verified =
🤔
Do you have the openssl
package installed? do you see any openssl
issues in the logs?
This is the official docker image, so what I see for openssl version is: OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)
Not seeing anything in the logs...but I am happy to test the image more if you have any suggestions
OK!
Can you also send me the $headers
array from here: https://github.com/Automattic/wordpress-activitypub/blob/57299fc131dfff8b72846224257525c88b23569b/includes/class-signature.php#L261
I did a quick openssl s_client test against the site from the image and it worked OK...so that is a good sign.
[Wed Sep 04 15:00:54.061856 2024] [php:notice] [pid 35:tid 35] [client 2620:128:2000:254::80:56700] Activitypub\Signature::verify_http_signature: headers = Array\n(\n [authorization] => Array\n (\n [0] => \n )\n\n [host] => Array\n (\n [0] => sydneyschwarz.breepriv.lothlorien.net\n )\n\n [x_request_id] => Array\n (\n [0] => bc0b04932eeb12c051daf35597522e5c\n )\n\n [x_real_ip] => Array\n (\n [0] => 2620:128:2000:254:de3e:b475:eef6:27dd\n )\n\n [x_forwarded_for] => Array\n (\n [0] => 2620:128:2000:254:de3e:b475:eef6:27dd\n )\n\n [x_forwarded_host] => Array\n (\n [0] => sydneyschwarz.breepriv.lothlorien.net\n )\n\n [x_forwarded_port] => Array\n (\n [0] => 443\n )\n\n [x_forwarded_proto] => Array\n (\n [0] => https\n )\n\n [x_forwarded_scheme] => Array\n (\n [0] => https\n )\n\n [x_scheme] => Array\n (\n [0] => https\n )\n\n [x_original_forwarded_for] => Array\n (\n [0] => 2620:128:2000:0:250:56ff:fe95:3ffd\n )\n\n [content_length] => Array\n (\n [0] => 249\n )\n\n [user_agent] => Array\n (\n [0] => http.rb/5.1.1 (Mastodon/4.2.12; +https://mastodon.lothlorien.net/)\n )\n\n [date] => Array\n (\n [0] => Wed, 04 Sep 2024 15:00:53 GMT\n )\n\n [accept_encoding] => Array\n (\n [0] => gzip\n )\n\n [digest] => Array\n (\n [0] => SHA-256=LCImew+Ldr8+vM1Ikl4nP1IUTkDG9LzXfsWJnWabxAs=\n )\n\n [content_type] => Array\n (\n [0] => application/activity+json\n )\n\n [signature] => Array\n (\n [0] => keyId="https://mastodon.lothlorien.net/users/jason#main-key",algorithm="rsa-sha256",headers="(request-target) host date digest content-type",signature="ZIS55PWqhTNMiru2+XDy0XpDjkIyAY7FCwroc1SZ6g5WLG+1zXtXmJgmRZ1rTZ9sa/bH/56RWzKKQDKBNBaj5KuTIB+2vIZ2j5rq32jVr93Bz24lU0SlkLYVkqhmLVpEauvf68/PXE5Kvjeax6/FKUup3AHrgQAFg0BY7mi/EoRT8MlvLRgGTYGFFQ7KXr/gcCqM1jGUr24eR+czArV31jTtxMePXunduuS7fuLInJLZ4eXf46hy9gEOnfLMzjlCAH+m9pryGXQeEbUcayacW93EceZpj/W2RaqwZhryrT4vOmfDZ+JZnp/1/Qlc1saKFbyKn48DEE0YnW8kbAI5Dg=="\n )\n\n [x_forwarded_server] => Array\n (\n [0] => sydneyschwarz.com\n )\n\n [(request-target)] => Array\n (\n [0] => post /wp-json/activitypub/1.0/actors/1/inbox\n )\n\n)\n
OK, that is interesting...it is grabbing an internal hostname in the headers
Would having the internal host name be a problem? It is using an Apache reverse proxy in front of the internal host which is running in a private kubernetes cluster.
yes, because the remote server generates the signature with the public host and your blog tries to verify with the internal host... or the other way around... that's why it might mismatch!
you can try to add $headers['host'] = array( 'lothlorien.net' );
before https://github.com/Automattic/wordpress-activitypub/blob/57299fc131dfff8b72846224257525c88b23569b/includes/class-signature.php#L468
Maybe you can also log $signed_data
to see if the host is set properly through the hack!
or $headers['host'] = array( 'sydneyschwarz.com' );
or whatever the "real" domain is :)
OK, so that seems to be it:
[Wed Sep 04 15:25:05.644275 2024] [php:notice] [pid 34:tid 34] [client 2620:128:2000:254:de3e:b475:eef6:27dd:0] Activitypub\Signature::verify_http_signature: verified = 1
So now the deeper question: Should we be able to set the host name just like the siteurl in the wp_options table somehow?
Or should it use a parsed version of the siteurl since that is what wp-admin is also using?
yes, maybe checking the siteurl
against the headers might be a good idea or to directly replace the header with the siteurl
.
Let me think about the implications that the change might have!
@jjschwarz can you maybe file an issue describing this very specific issue?
Then we can close this one?
Even better, now that I know the cause I have modified the Apache to solve the issue. The "ProxyPreserveHost On" option will let us get around the issue. In some setups that may also require the "SSLProxyCheckPeerName off" option if the back end server does not also respond to the same host name for SSL. Not sure if you want to add that to any documentation regarding reverse proxy setups with this plugin.
Thank you again for the debugging!
No problem!
Are you maybe willing to write an FAQ entry?
I also try to add this case to the health check tests. What might be the best PHP function that might always return the local domain instead of the public one?
That is the million dollar question...because the only thing I can find is the manually configured siteurl variable.
I would be happy to write up a brief FAQ. Do you want me to post it in this thread?
That is the million dollar question...because the only thing I can find is the manually configured siteurl variable.
No, I mean is there a way to retrieve the internal one? So that I have something that I can compare with the siteurl
?!?
I would be happy to write up a brief FAQ. Do you want me to post it in this thread?
Either that or directly a PR if you would love to!
So the only place I am seeing that is in the $headers['host'] variable you are getting from the upstream server, since it is not the actual host name of the pod itself in K8S, but the name of the ingress, which the web server can't even see directly.
What file should I target the PR against? I am not seeing any general FAQ in the project folder.
Here is the FAQ: https://github.com/Automattic/wordpress-activitypub/blob/57299fc131dfff8b72846224257525c88b23569b/readme.txt#L53
Can you maybe revert your Apache changes temporarily and post me the result of $_SERVER['SERVER_NAME']
or check if one of the other entries have the wrong host?
Any thoughts on which file I should debug that in for you? I wasn't sure if you wanted it from wp-login.php or another place?
I tried it from wp-login.php and the variable tracks what I send over from the proxy. Without my change: [Wed Sep 04 16:48:49.769625 2024] [php:notice] [pid 21:tid 21] [client 2620:128:2000:254::80:34380] Server name sydneyschwarz.breepriv.lothlorien.net
with my change: [Wed Sep 04 16:47:42.900227 2024] [php:notice] [pid 52:tid 52] [client 2620:128:2000:254:de3e:b475:eef6:27dd:0] Server name sydneyschwarz.com
So if that mismatches the siteurl then it would appear you will have issues.
Nice, thanks! Then I will prepare a check for that!
Thanks a lot for all your help with the debugging!
No problem. I will write up something for the FAQ after my meetings settle down today. Hopefully that will help others that use reverse proxies avoid this issue. We are slowly migrating our installations from running right on the web server to running in Kubernetes with the web server being the reverse proxy to keep down issues we have been in the past. Hopefully the results are fairly clean once it is all done.
Pull request 888 created with some basic documentation
Quick summary
I am suffering the same issue as case 591 (https://github.com/Automattic/wordpress-activitypub/issues/591) on a brand new deployment of WordPress from the 6.6 Docker image. I created the site, installed the plugin and tested that sydneyschwarz.com@sydneyschwarz.com was resolving with webfinger. I am able to find the site from the Mastodon server, however when I attempt to follow I get stuck in the pending state forever.
Steps to reproduce
What you expected to happen
I expect the follow to function and to show up on the follower list
What actually happened
Mastodon server states it can not send the request to the WordPress server and the follow is hung.
Impact
All
Available workarounds?
No and the platform is unusable
Logs or notes
I did check the settings to make sure the post format did not include index.php and that the .htaccess matched the other working sites.