codius + moneyd running on the client side - upload never works

theogravity commented 6 years ago

I finally managed to create my own sample app:

https://github.com/theogravity/codius-hello-world

and I'm able to run the manifest on the machine where the Codius server runs, but I'm unable to run it from my laptop, where I have moneyd running (not on testnet).

The codius execution just stalls after a certain point - unsure what's the issue.

I've attached logs of the codius execution and a snippet from moneyd

codius-client.txt moneyd.txt

retryW commented 6 years ago

Strange as I can do it flawlessly from my laptop (and Windows for that matter)

D:\Users\Administrator>codius upload D:\temp\test-react.json --host https://host1.codius.live --duration 300
Successfully Uploaded Pods to:
[ { url: 'https://lg6gjhhh2b3if2tzpbms6cr3hdmh7dixa2v4an6pz4tdxdmqtjpa.host1.codius.live/',
    manifestHash: 'lg6gjhhh2b3if2tzpbms6cr3hdmh7dixa2v4an6pz4tdxdmqtjpa',
    host: 'https://host1.codius.live',
    expiry: '2018-06-18T03:59:25.736Z',
    expirationDate: '06-18-2018 1:59:25 +1000',
    expires: 'in 5 minutes',
    pricePaid: '1141' } ]

D:\Users\Administrator>codius upload D:\temp\test-react.json --host https://codius-1.logancodius.xyz --duration 300
Successfully Uploaded Pods to:
[ { url: 'https://lg6gjhhh2b3if2tzpbms6cr3hdmh7dixa2v4an6pz4tdxdmqtjpa.codius-1.logancodius.xyz/',
    manifestHash: 'lg6gjhhh2b3if2tzpbms6cr3hdmh7dixa2v4an6pz4tdxdmqtjpa',
    host: 'https://codius-1.logancodius.xyz',
    expiry: '2018-06-18T04:42:01.657Z',
    expirationDate: '06-18-2018 2:42:01 +1000',
    expires: 'in 5 minutes',
    pricePaid: '1141' } ]

theogravity commented 6 years ago

Can you describe your setup? Do you only have moneyd running on your machine?

Are you running your example from the Codius host itself, or from another machine? I'm able to get the examples working from the Codius host itself, but not outside of it.

We shouldn't need anything other than moneyd + codius client (eg no hyperd, codiusd) to upload a manifest to a remote host

retryW commented 6 years ago

I'm running a Windows 7 laptop (not a codius host).

I installed NodeJS from their official downloads page.
npm install -g moneyd moneyd-uplink-xrp
moneyd xrp:configure --advanced # Had 16 extra XRP in wallet so could setup with same wallet as my codius host
npm install -g codius
moneyd xrp:start # leave in background and open separate console

Also make sure moneyd has actually finished connecting before running the codius CLI cmds

2018-06-19T07:44:02.129Z connector:app info connector ready (republic attitude). address=g.scylla.client.client1

Then I created a manifest json file on my machine and run the following upload command (I'll include the output too)

D:\Users\Administrator>codius upload D:\temp\test-react.json --host https://host1.codius.live --duration 400
Successfully Uploaded Pods to:
[ { url: 'https://lg6gjhhh2b3if2tzpbms6cr3hdmh7dixa2v4an6pz4tdxdmqtjpa.host1.codius.live/',
    manifestHash: 'lg6gjhhh2b3if2tzpbms6cr3hdmh7dixa2v4an6pz4tdxdmqtjpa',
    host: 'https://host1.codius.live',
    expiry: '2018-06-19T07:52:33.786Z',
    expirationDate: '06-19-2018 5:52:33 +1000',
    expires: 'in 6 minutes',
    pricePaid: '1522' } ]

D:\Users\Administrator>

sharafian commented 6 years ago

This appears to be an issue depending on the host to whom you upload. For certain hosts, it looks like their parent connector is exceeding the maximum balance to them, which is blocking further traffic. It's strange, though, because the connector should be sending a payment channel claim to the moneyd instance after almost every packet. I'll look into it from the connector side, though.

kruisdraad commented 6 years ago

@sharafian i think i have the save issue, weirdly i cannot understand the logic on it. We have deployed 4 identical machines, with same installations. Only things that are different are: hostname, ip, ssl cert, XRP secret (diff wallets as per manual requirement).

codius3.mlab.company: works, and fast codius2.mlab.company: works, but takes about 30-90 secs to load codius1.mlab.company: just endless loading, 10 min timeout codius4.mlab.company: just endless loading, 10 min timeout

These results are tested with https://upload.host1.codius.live/ and you keep getting the same results on timing/response even across reboots, restarts, etc on our side. Load times exactly the same. Initial tests with Win10+Chrome then we moved to another system (Win2k8+IE) and again ... exactly the same responses.

I have looked at the TCPdumps and decrypted requests and responses. The working systems get a first request and then reply with a 402 payment required. A bit later (timing above table) the request is doing again on nodes 3 and 2 replied with a 200 OK starting the contract on the server. Servers 1 and 4 never get the 2nd request at all. Because we captured the request we can do the 2nd request manually and interesting it give the error: {"statusCode":402,"error":"Payment Required","message":"Failed to get payment before timeout"}

Arround the same time we get an error in syslog: connector:alert-middleware warn generating alert for account=parent triggeredBy=g.scylla message="exceeded maximum balance."

Interesting fact the servers can online in the order: codius3, codius2, codius4, codius1 and the first box that came online was the fastest. It might not be related, but it seems like worth mentioning.

As we are able to reproduce this consistently we'd be happy to help with debugging this.

kruisdraad commented 6 years ago

Also seeing a lot of debug errors on all 4 servers (perhaps known issue and not related):

Jul 15 12:24:39 codius4 moneyd[1666]: 2018-07-15T10:24:39.212Z connector:error-handler-middleware[local] debug error in data handler, creating rejection. ilpErrorCode=F00 error=BadRequestError: cannot process route control messages from non-peers. Jul 15 12:24:39 codius4 moneyd[1666]: at RouteBroadcaster.handleRouteControl (/usr/lib/node_modules/moneyd/node_modules/ilp-connector/src/services/route-broadcaster.ts:209:13) Jul 15 12:24:39 codius4 moneyd[1666]: at CcpController.handleRouteControl (/usr/lib/node_modules/moneyd/node_modules/ilp-connector/src/controllers/ccp.ts:40:27) Jul 15 12:24:39 codius4 moneyd[1666]: at CcpController.handle (/usr/lib/node_modules/moneyd/node_modules/ilp-connector/src/controllers/ccp.ts:28:21) Jul 15 12:24:39 codius4 moneyd[1666]: at PeerProtocolController.handle (/usr/lib/node_modules/moneyd/node_modules/ilp-connector/src/controllers/peer-protocol.ts:30:33) Jul 15 12:24:39 codius4 moneyd[1666]: at IlpPrepareController.sendData (/usr/lib/node_modules/moneyd/node_modules/ilp-connector/src/controllers/ilp-prepare.ts:40:42) Jul 15 12:24:39 codius4 moneyd[1666]: at Core.processData (/usr/lib/node_modules/moneyd/node_modules/ilp-connector/src/services/core.ts:44:42) Jul 15 12:24:39 codius4 moneyd[1666]: at handleData (/usr/lib/node_modules/moneyd/node_modules/ilp-connector/src/services/middleware-manager.ts:183:65) Jul 15 12:24:39 codius4 moneyd[1666]: at dispatch (/usr/lib/node_modules/moneyd/node_modules/ilp-connector/src/lib/utils.ts:78:14) Jul 15 12:24:39 codius4 moneyd[1666]: at next (/usr/lib/node_modules/moneyd/node_modules/ilp-connector/src/lib/utils.ts:79:16) Jul 15 12:24:39 codius4 moneyd[1666]: at /usr/lib/node_modules/moneyd/node_modules/ilp-connector/src/middlewares/stats.ts:50:34 Jul 15 12:24:41 codius4 moneyd[1666]: 2018-07-15T10:24:41.457Z ilp-xrp-channel-watcher checking 1 channels for expiry

kruisdraad commented 6 years ago

Interesting fact both broken servers use btp.connector0.com as a server, the slow server uses ilsp.openafricanetwork.org which connects bad for me from the Netherlands (high latency). The working server uses client.scyl.la. This setting seems hardcoded in the configfile and never changes ? (if thats the case perhaps add 3 and do round-robin or something, less chance for a now to break down on this SPOF?)

kruisdraad commented 6 years ago

@theogravity can you check your .moneyd.json file under options->server and then after the @ sign which server you are using and post it there?

follow https://github.com/codius/codiusd/wiki/Troubleshooting#creating-a-new-channel-with-moneyd and remove the channel, wait an hour and continue with a new config. When configuring try not to use btp.connector0.com as a server as thats the one blocking me it seems.

codius / old-codius-cli

codius + moneyd running on the client side - upload never works #41