search

skohub-io / skohub-pubsub

Apache License 2.0
15 stars 3 forks source link

Messages to inbox are not forwarded to followers #40

Open acka47 opened 3 years ago

acka47 commented 3 years ago

Whether I use the editor or curl, I can post to the inbox but it won't be forwarded to the followers' inboxes.

Curl example:

$ curl -vH "content-type: application/json" 'https://skohub.io/inbox?actor=dini-ag-kim%2Fhochschulfaechersystematik%2Fheads%2Fmaster%2Fw3id.org%2Fkim%2Fhochschulfaechersystematik%2Fn271' --data-binary '{"@context":["https://w3id.org/kim/lrmi-profile/draft/context.jsonld",{"@language":"de"}],"name":"222Test-Deutsch als Zweitsprache","id":"https://unterrichten.zum.de/wiki/Deutsch_als_Zweitsprache","description":"Hier ein Versuch, einige markante Unterschiede in wenigen Worten darzustellen: Der entscheidende Unterschied ist die Spracherwerbssituation:","about":[{"id":"https://w3id.org/kim/hochschulfaechersystematik/n271","prefLabel":{"de":"Deutsch als Fremdsprache oder als Zweitsprache"},"type":"Concept","inScheme":{"id":"https://w3id.org/kim/hochschulfaechersystematik/scheme"}}]}'
*   Trying 193.30.112.187...
* TCP_NODELAY set
* Connected to skohub.io (193.30.112.187) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=DE; ST=Nordrhein-Westfalen; L=Koeln; O=Hochschulbibliothekszentrum des Landes Nordrhein-Westfalen; OU=Rechenzentrum; CN=www.skohub.io
*  start date: May 13 06:41:09 2020 GMT
*  expire date: Aug 15 06:41:09 2022 GMT
*  subjectAltName: host "skohub.io" matched cert's "skohub.io"
*  issuer: C=DE; O=Verein zur Foerderung eines Deutschen Forschungsnetzes e. V.; OU=DFN-PKI; CN=DFN-Verein Global Issuing CA
*  SSL certificate verify ok.
> POST /inbox?actor=dini-ag-kim%2Fhochschulfaechersystematik%2Fheads%2Fmaster%2Fw3id.org%2Fkim%2Fhochschulfaechersystematik%2Fn271 HTTP/1.1
> Host: skohub.io
> User-Agent: curl/7.58.0
> Accept: */*
> content-type: application/json
> Content-Length: 586
> 
* upload completely sent off: 586 out of 586 bytes

The command will stop at this point for quite some time and after a while this is added:

< HTTP/1.1 404 Not Found
< Date: Wed, 14 Oct 2020 13:15:31 GMT
< Server: Apache/2.4.38 (Debian)
< Content-Length: 275
< Content-Type: text/html; charset=iso-8859-1
< 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
<hr>
<address>Apache/2.4.38 (Debian) Server at 193.30.112.82 Port 80</address>
</body></html>
* Connection #0 to host skohub.io left intact

Result: the message is in the inbox but I do not get it via Mastodon although I am a follower of the respective concept.

Using the editor or the extension it is very similar:

image

@dr0i, there must be something wrong on the server side, I guess. It did function before, though. Maybe @literarymachine has an idea...

literarymachine commented 3 years ago

I believe this is the relevant message:

ALPN, server did not agree to a protocol

This could well be something in the proxy config.

acka47 commented 3 years ago

On the test server, it doesn't work as well:

$ curl -vH "content-type: application/json" 'https://test.skohub.io/inbox?actor=hbz%2Fvocabs-edu%2Fheads%2Fmaster%2Fw3id.org%2Fclass%2Fesc%2Fn0322' --data-binary '{"@context":["https://w3id.org/kim/lrmi-profile/draft/context.jsonld",{"@language":"de"}],"name":"222Test-Bibliaries","id":"https://unterrichten.zum.de/wiki/Deutsch_als_Zweitsprache","description":"Hier ein Versuch, einige markante Unterschiede in wenigen Worten darzustellen: Der entscheidende Unterschied ist die Spracherwerbssituation:","about":[{"id":"https://test.skohub.io/hbz/vocabs-edu/heads/master/w3id.org/class/esc/n0322","prefLabel":{"de":"Deutsch als Fremdsprache oder als Zweitsprache"},"type":"Concept","inScheme":{"id":"https://w3id.org/kim/hochschulfaechersystematik/scheme"}}]}'
*   Trying 193.30.112.187...
* Connected to test.skohub.io (193.30.112.187) port 443 (#0)
* found 127 certificates in /etc/ssl/certs/ca-certificates.crt
* found 520 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*    server certificate verification OK
*    server certificate status verification SKIPPED
*    common name: www.skohub.io (matched)
*    server certificate expiration date OK
*    server certificate activation date OK
*    certificate public key: RSA
*    certificate version: #3
*    subject: C=DE,ST=Nordrhein-Westfalen,L=Koeln,O=Hochschulbibliothekszentrum des Landes Nordrhein-Westfalen,OU=Rechenzentrum,CN=www.skohub.io
*    start date: Wed, 13 May 2020 06:41:09 GMT
*    expire date: Mon, 15 Aug 2022 06:41:09 GMT
*    issuer: C=DE,O=Verein zur Foerderung eines Deutschen Forschungsnetzes e. V.,OU=DFN-PKI,CN=DFN-Verein Global Issuing CA
*    compression: NULL
* ALPN, server did not agree to a protocol
> POST /inbox?actor=hbz%2Fvocabs-edu%2Fheads%2Fmaster%2Fw3id.org%2Fclass%2Fesc%2Fn0322 HTTP/1.1
> Host: test.skohub.io
> User-Agent: curl/7.47.0
> Accept: */*
> content-type: application/json
> Content-Length: 595
> 
* upload completely sent off: 595 out of 595 bytes
< HTTP/1.1 404 Not Found
< Date: Thu, 15 Oct 2020 08:39:24 GMT
< Server: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1
< Content-Length: 196
< Content-Type: text/html; charset=iso-8859-1
< 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>
* Connection #0 to host test.skohub.io left intact
dr0i commented 3 years ago

If you use without SSL, it seems to work, right?

acka47 commented 3 years ago

If you use without SSL, it seems to work, right?

Nope, when using http-only, the server sends a 301 redirect to the https version:

$ curl -vH "content-type: application/json" 'http://skohub.io/inbox?actor=dini-ag-kim%2Fhochschulfaechersystematik%2Fheads%2Fmaster%2Fw3id.org%2Fkim%2Fhochschulfaechersystematik%2Fn271' --data-binary '{"@context":["https://w3id.org/kim/lrmi-profile/draft/context.jsonld",{"@language":"de"}],"name":"222Test-Deutsch als Zweitsprache","id":"https://unterrichten.zum.de/wiki/Deutsch_als_Zweitsprache","description":"Hier ein Versuch, einige markante Unterschiede in wenigen Worten darzustellen: Der entscheidende Unterschied ist die Spracherwerbssituation:","about":[{"id":"https://w3id.org/kim/hochschulfaechersystematik/n271","prefLabel":{"de":"Deutsch als Fremdsprache oder als Zweitsprache"},"type":"Concept","inScheme":{"id":"https://w3id.org/kim/hochschulfaechersystematik/scheme"}}]}'
*   Trying 193.30.112.187...
* Connected to skohub.io (193.30.112.187) port 80 (#0)
> POST /inbox?actor=dini-ag-kim%2Fhochschulfaechersystematik%2Fheads%2Fmaster%2Fw3id.org%2Fkim%2Fhochschulfaechersystematik%2Fn271 HTTP/1.1
> Host: skohub.io
> User-Agent: curl/7.47.0
> Accept: */*
> content-type: application/json
> Content-Length: 586
> 
* upload completely sent off: 586 out of 586 bytes
< HTTP/1.1 301 Moved Permanently
< Date: Thu, 15 Oct 2020 09:45:19 GMT
< Server: Apache/2.4.10 (Linux/SUSE)
< Location: https://skohub.io/inbox?actor=dini-ag-kim%2Fhochschulfaechersystematik%2Fheads%2Fmaster%2Fw3id.org%2Fkim%2Fhochschulfaechersystematik%2Fn271
< Content-Length: 427
< Content-Type: text/html; charset=iso-8859-1
< 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://skohub.io/inbox?actor=dini-ag-kim%2Fhochschulfaechersystematik%2Fheads%2Fmaster%2Fw3id.org%2Fkim%2Fhochschulfaechersystematik%2Fn271">here</a>.</p>
<hr>
<address>Apache/2.4.10 (Linux/SUSE) Server at skohub.io Port 80</address>
</body></html>
* Connection #0 to host skohub.io left intact
dr0i commented 3 years ago

Interesting. Use the "-L" with curl and it seems to work, no?

dr0i commented 3 years ago

Uh, right: "POST has been converted to GET and no more data is forwarded." , so this is a different thing.

dr0i commented 3 years ago

Ok. Got it:

  1. HD space was low (<5 GB), elasticsearch complains. This was passed to the skohub-pubsub.log:

(node:12036) UnhandledPromiseRejectionWarning: ResponseError: cluster_block_exception at IncomingMessage. (/home/lod/git/skohub-pubsub/node_modules/@elastic/elasticsearch/lib/Transport.js:294:25)

Fixed this by freeing disk space. Restarted ES.

  1. After fixing 1. and POSTing the data ES responds with:

MapperParsingException: object mapping for [attachment.@context] tried to parse field [null] as object, but found a concrete value

Manipulating the data to POST to:

  "@context" :{ 
     "id" : "https://w3id.org/kim/lrmi-profile/draft/context.jsonld"
},
...

the document is created.

To not happen 1. again I will put free space observated by Nagios which warns us by email. To fix 2. : you may want to have a language attribute for @context even if this does not make sense (or does it?) but we have to tweak the ES settings to allow this. (would this be compatible with the already indexed data?)

dr0i commented 3 years ago

fyi, index was set to "read only" because of the low watermark of disk space since 2020-10-03.

dr0i commented 3 years ago

As discussed offline with @acka47 we delete the ES index. A new indexed document would then determine the ES settings field structure to be ok with what is used atm.

dr0i commented 3 years ago

Not working. After deleting and recreation with the json in question, ES logs:

java.lang.IllegalArgumentException: Can't merge a non object mapping [attachment.@context] with an object mapping [attachment.@context]

dr0i commented 3 years ago

@acka47 this would work :

{"@context":[ {"@id" : "https://w3id.org/kim/lrmi-profile/draft/context.jsonld"},{"@language":"de"}],

we are ok with that?

dr0i commented 3 years ago

@acka47 confirmed to change the schema at https://github.com/dini-ag-kim/lrmi-profile to comply to elasticsearch's "Arrays with a mixture of data types are not supported".

acka47 commented 3 years ago

@acka47 confirmed to change the schema at https://github.com/dini-ag-kim/lrmi-profile to comply to elasticsearch's "Arrays with a mixture of data types are not supported".

See https://github.com/dini-ag-kim/lrmi-profile/issues/26

dr0i commented 3 years ago

Deployed to test. Please test it @acka47, use the 'test.skohub.io' URL to do so.

acka47 commented 3 years ago

It does work better than before but the messages won't get trough to Mastodon anyway. I created a new build at https://test.skohub.io/build/?id=e06c1fe4-a93f-4a92-bed7-61e70a7eabd1.

I followed n271, see the followers list. I posted with curl like this:

$ curl -vH "content-type: application/json" 'https://test.skohub.io/inbox?actor=acka47%2Fhochschulfaechersystematik%2Fheads%2Fmaster%2Fw3id.org%2Fkim%2Fhochschulfaechersystematik%2Fn271' --data-binary '{"@context":["https://w3id.org/kim/lrmi-profile/draft/context.jsonld",{"@language":"de"}],"name":"222Test-Deutsch als Zweitsprache","id":"https://unterrichten.zum.de/wiki/Deutsch_als_Zweitsprache","description":"Hier ein Versuch, einige markante Unterschiede in wenigen Worten darzustellen: Der entscheidende Unterschied ist die Spracherwerbssituation:","about":[{"id":"https://w3id.org/kim/hochschulfaechersystematik/n271","prefLabel":{"de":"Deutsch als Fremdsprache oder als Zweitsprache"},"type":"Concept","inScheme":{"id":"https://w3id.org/kim/hochschulfaechersystematik/scheme"}}]}'
*   Trying 193.30.112.187...
* Connected to test.skohub.io (193.30.112.187) port 443 (#0)
* found 127 certificates in /etc/ssl/certs/ca-certificates.crt
* found 520 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*    server certificate verification OK
*    server certificate status verification SKIPPED
*    common name: www.skohub.io (matched)
*    server certificate expiration date OK
*    server certificate activation date OK
*    certificate public key: RSA
*    certificate version: #3
*    subject: C=DE,ST=Nordrhein-Westfalen,L=Koeln,O=Hochschulbibliothekszentrum des Landes Nordrhein-Westfalen,OU=Rechenzentrum,CN=www.skohub.io
*    start date: Wed, 13 May 2020 06:41:09 GMT
*    expire date: Mon, 15 Aug 2022 06:41:09 GMT
*    issuer: C=DE,O=Verein zur Foerderung eines Deutschen Forschungsnetzes e. V.,OU=DFN-PKI,CN=DFN-Verein Global Issuing CA
*    compression: NULL
* ALPN, server did not agree to a protocol
> POST /inbox?actor=acka47%2Fhochschulfaechersystematik%2Fheads%2Fmaster%2Fw3id.org%2Fkim%2Fhochschulfaechersystematik%2Fn271 HTTP/1.1
> Host: test.skohub.io
> User-Agent: curl/7.47.0
> Accept: */*
> content-type: application/json
> Content-Length: 586
> 
* upload completely sent off: 586 out of 586 bytes
< HTTP/1.1 201 Created
< Date: Thu, 29 Oct 2020 09:45:23 GMT
< Server: Apache/2.4.10 (Linux/SUSE)
< X-Powered-By: Express
< Access-Control-Allow-Origin: *
< Location: https://test.skohub.io/m/5a0776c9-9856-4ccf-9ecd-9c89435f673e
< Content-Length: 0
< 
* Connection #0 to host test.skohub.io left intact

However, the message is not forwarded to the followers it seems as I don't get a message in Mastodon. One strange thing in this context: the concept is not listed in my "following" list on Mastodon (but others are):

Bildschirmfoto vom 2020-10-29 10-51-21

acka47 commented 3 years ago

However, as the original problem seems to be fixed, we can already deploy the changes to production and then look into the remaining issue.

dr0i commented 3 years ago

Deployed to production. Please test it @acka47 .

acka47 commented 3 years ago

I am sorry to say that but it works on production:

Bildschirmfoto vom 2020-10-29 15-26-30

dr0i commented 3 years ago

Comparing the logs of production (it's just empty) and the one I got on test.skohub:

(node:24850) UnhandledPromiseRejectionWarning: Error: Unauthorized at Request.callback (/home/lod/git/skohub-pubsub/node_modules/superagent/lib/node/index.js:804:15) at IncomingMessage. (/home/lod/git/skohub-pubsub/node_modules/superagent/lib/node/index.js:1036:18) at Stream.emit (events.js:311:20) at Unzip. (/home/lod/git/skohub-pubsub/node_modules/superagent/lib/node/unzip.js:55:12) at Unzip.emit (events.js:323:22) at endReadableNT (_stream_readable.js:1204:12) at processTicksAndRejections (internal/process/task_queues.js:84:21) (node:24850) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 14)

acka47 commented 3 years ago

@literarymachine, does this maybe ring a bell? If not, we can leave this for now nbecause it works on production where it is most important...

literarymachine commented 3 years ago

UnhandledPromiseRejectionWarning: Error: Unauthorized

It could be that the message's signature cannot be verified, you might want to compare the configurations on test and prod starting here:

https://github.com/skohub-io/skohub-vocabs/blob/master/.env.example#L6 https://github.com/skohub-io/skohub-pubsub/blob/master/src/activitypub.js#L19

Unfortunately I don't really have the time right now to wrap my head back around those authentication issues...

dr0i commented 3 years ago

PUBLIC_KEY on both machines are exactly the same. npm test on both machines (shutting down the running daemon before because the test is using the same port as the inbuild-webHookServer test) return succesfully. Is there a private key involved and are these not the same for test and production?