No new contracts despite FormContract calls

[mtlynch]

I'm running a host and I was getting new contracts for the first few days (while still on 0.6.0), but then it seems to have stopped. I upgraded to 1.0.0 and it doesn't seem to have made any difference. I haven't had any increase in host storage in days, but my FormContract calls keep increasing, which I assume to mean people are trying to form contracts with my host.

$ siac host -v
General Info:
        Estimated Competitive Price: 1.547 KS

Host Internal Settings:
        acceptingcontracts:   Yes
        maxduration:          25 Weeks
        maxdownloadbatchsize: 17.83 MB
        maxrevisebatchsize:   17.83 MB
        netaddress:           [redacted]
        windowsize:           0 Weeks

        collateral:       25 KS / TB / Month
        collateralbudget: 5 MS
        maxcollateral:    250 KS Per Contract

        mincontractprice:         1 SC
        mindownloadbandwithprice: 5 KS / TB
        minstorageprice:          1.4 KS / TB / Month
        minuploadbandwidthprice:  100 SC / TB

Host Financials:
        Contract Count:               0
        Transaction Fee Compensation: 0 H
        Transaction Fee Expenses:     0 H

        Storage Revenue:           0 H
        Potential Storage Revenue: 5.245 SC

        Locked Collateral: 19.61 KS
        Risked Collateral: 131.1 SC
        Lost Collateral:   0 H

        Download Revenue:           0 H
        Potential Download Revenue: 0 H
        Upload Revenue :            0 H
        Potential Upload Revenue:   404.8 mS

RPC Stats:
        Error Calls:        7780
        Unrecognized Calls: 95
        Download Calls:     1
        Renew Calls:        14
        Revise Calls:       6895
        Settings Calls:     52359
        FormContract Calls: 932

Storage Folders:
    Used        Capacity     % Used    Path
    4.064 GB    1.0000 TB    0.41      /mnt/sia/host-storage

Then a few hours later

$ siac host -v
General Info:
        Estimated Competitive Price: 1.198 KS

Host Internal Settings:
        acceptingcontracts:   Yes
        maxduration:          25 Weeks
        maxdownloadbatchsize: 17.83 MB
        maxrevisebatchsize:   17.83 MB
        netaddress:           [redacted]
        windowsize:           0 Weeks

        collateral:       25 KS / TB / Month
        collateralbudget: 5 MS
        maxcollateral:    250 KS Per Contract

        mincontractprice:         1 SC
        mindownloadbandwithprice: 5 KS / TB
        minstorageprice:          1.4 KS / TB / Month
        minuploadbandwidthprice:  100 SC / TB

Host Financials:
        Contract Count:               0
        Transaction Fee Compensation: 0 H
        Transaction Fee Expenses:     0 H

        Storage Revenue:           0 H
        Potential Storage Revenue: 5.245 SC

        Locked Collateral: 19.61 KS
        Risked Collateral: 131.1 SC
        Lost Collateral:   0 H

        Download Revenue:           0 H
        Potential Download Revenue: 0 H
        Upload Revenue :            0 H
        Potential Upload Revenue:   404.8 mS

RPC Stats:
        Error Calls:        7907
        Unrecognized Calls: 95
        Download Calls:     1
        Renew Calls:        14
        Revise Calls:       6898
        Settings Calls:     53947
        FormContract Calls: 1056

Storage Folders:
    Used        Capacity     % Used    Path
    4.064 GB    1.0000 TB    0.41      /mnt/sia/host-storage

I also notice that the host price keeps dropping so is it just that I'm priced too high?

[Puychack]

Same problem for me. 700 MB with cli 0.6.0 and not,a single byte since cli 1.0.0 despite many settings calls. My collateral is set to ~1.2kS

\ Edit ** I've found the problem for me : default values ate up all my collateral budget faster than I thought so I couldn't get more contracts.

[DavidVorick]

Make sure your wallet is unlocked, and make sure that you have collateral posted.

[mtlynch]

It looks like I have enough collateral. Any other ideas?

$ siac wallet && siac host -v
Wallet status:
Encrypted, Unlocked
Confirmed Balance:   508.2 KS
Unconfirmed Delta:  +0 H
Exact:               508208289674835634184544321536 H
Siafunds:            0 SF
Siafund Claims:      0 H
General Info:
        Estimated Competitive Price: 3.983 KS

Host Internal Settings:
        acceptingcontracts:   Yes
        maxduration:          25 Weeks
        maxdownloadbatchsize: 17.83 MB
        maxrevisebatchsize:   17.83 MB
        netaddress:           [redacted]
        windowsize:           0 Weeks

        collateral:       25 KS / TB / Month
        collateralbudget: 5 MS
        maxcollateral:    250 KS Per Contract

        mincontractprice:         1 SC
        mindownloadbandwithprice: 5 KS / TB
        minstorageprice:          500 SC / TB / Month
        minuploadbandwidthprice:  100 SC / TB

Host Financials:
        Contract Count:               1
        Transaction Fee Compensation: 0 H
        Transaction Fee Expenses:     0 H

        Storage Revenue:           0 H
        Potential Storage Revenue: 5.245 SC

        Locked Collateral: 20.61 KS
        Risked Collateral: 131.1 SC
        Lost Collateral:   0 H

        Download Revenue:           0 H
        Potential Download Revenue: 0 H
        Upload Revenue :            0 H
        Potential Upload Revenue:   404.8 mS

RPC Stats:
        Error Calls:        9429
        Unrecognized Calls: 103
        Download Calls:     1
        Renew Calls:        14
        Revise Calls:       7039
        Settings Calls:     76306
        FormContract Calls: 2438

Storage Folders:
    Used        Capacity     % Used    Path
    4.064 GB    1.0000 TB    0.41      /mnt/sia/host-storage

[mtlynch]

I'm still seeing this on siad 1.01

[DavidVorick]

Acknowledged. Do you still only have one contract? The download price will make you unattractive on the new host, but your collateral is quite high and should ~approx make up for it in the current scoring system.

I've recently added a ton of logging to the host so I can follow all the errors in the form contract calls and the revise contract calls.

[mtlynch]

I've updated my settings:

General Info:
        Estimated Competitive Price: 849.4 SC

Host Internal Settings:
        acceptingcontracts:   Yes
        maxduration:          25 Weeks
        maxdownloadbatchsize: 17.83 MB
        maxrevisebatchsize:   17.83 MB
        netaddress:           [redacted]
        windowsize:           0 Weeks

        collateral:       25 KS / TB / Month
        collateralbudget: 5 MS
        maxcollateral:    250 KS Per Contract

        mincontractprice:         1 SC
        mindownloadbandwithprice: 1 KS / TB
        minstorageprice:          400 SC / TB / Month
        minuploadbandwidthprice:  90 SC / TB

Host Financials:
        Contract Count:               5
        Transaction Fee Compensation: 0 H
        Transaction Fee Expenses:     0 H

        Storage Revenue:           0 H
        Potential Storage Revenue: 5.245 SC

        Locked Collateral: 22.77 KS
        Risked Collateral: 131.1 SC
        Lost Collateral:   0 H

        Download Revenue:           0 H
        Potential Download Revenue: 0 H
        Upload Revenue :            0 H
        Potential Upload Revenue:   404.8 mS

RPC Stats:
        Error Calls:        19554
        Unrecognized Calls: 155
        Download Calls:     1
        Renew Calls:        30
        Revise Calls:       14688
        Settings Calls:     160045
        FormContract Calls: 4903

Storage Folders:
    Used        Capacity     % Used    Path
    4.064 GB    1.0000 TB    0.41      /mnt/sia/host-storage

Is there any way for me to see on my side why I'm not getting contracts? I just left most of the price settings at their defaults. I don't know how to evaluate other hosts to determine what's a competitive price for anything except storage.

[DavidVorick]

In terms of competitiveness, if you run siac host-v you'll be able to see where you fall on the chart. It's assembled randomly, but weighted according to host score. If you dig into modules/renter/hostdb/hostweight.go you can see what the equation is for scoring hosts, so you know what variables to focus on to improve your score.

Don't get too comfortable with it though, definitely still improving it. For example, there's no uptime/reliability consideration, and no Sybil attack mitigation yet. We'll need to drop both of those things in before the scoring starts getting really efficient. Not having them yet is one of the main reasons there aren't more tools for optimizing host score - they will be obsolete pretty quickly until all the base components are in place.

If you run the code in #1378, you'll get much better logging information. All FormContract calls that fail should have logged errors which pinpoint the exact part of the protocol where the failure occured. The fact that you have 4900 FormContract calls tells me that your host has a high score and is being selected often. Would be good to see what errors are preventing those calls from succeeding, though my guess is that it's something along the lines of ConsensusConflict: siacoin output does not exist, which I think is a problem on the renter's side.

[mtlynch]

Cool I'll keep an eye out for when #1378 gets merged

[mtlynch]

I installed the code that includes #1378 yesterday and I'm not seeing much in the logs:

2016/07/29 13:56:02.175760 announce.go:63: INFO: Successfully announced as [redacted]:9982 2016/08/01 10:38:27.722373 upnp.go:70: WARN: failed to discover external IP 2016/08/02 15:43:09.198886 upnp.go:70: WARN: failed to discover external IP 2016/08/03 12:16:25.955834 upnp.go:70: WARN: failed to discover external IP 2016/08/03 14:46:58.769884 upnp.go:70: WARN: failed to discover external IP 2016/08/03 23:02:39.262225 dependencies.go:115: STARTUP: Logging has started. 2016/08/03 23:02:39.477680 persist.go:181: WARN: NetAddress '' loaded from persist is invalid: missing port in address 2016/08/03 23:02:43.709998 network.go:74: ERROR: failed to forward port: no UPnP-enabled gateway found 2016/08/04 12:42:06.459114 storageobligations.go:558: Missed storage proof. Revenue would have been 311650144005677978470055936.

It seems to be having trouble with my network, but my port 9982 is open through all my firewalls.

[DavidVorick]

if you don't have upnp, the upnp code is going to fail, I don't think that's the issue.

If there are formcontract calls failing, they don't seem to be in the logs you've posted above.

[mtlynch]

Ah, whoops, I didn't realize that when you rebuild a Docker image with go get ..., it caches the response on subsequent builds, so I wasn't actually running the latest code. I've rebuilt the image with --no-cache and will update when I have logs.

[mtlynch]

Okay, there we go. I'm seeing logs now. The errors look like the following:

2016/08/07 23:10:30.543188 storageobligations.go:383: Failed to add storage obligation, transaction set was not accepted: consensus conflict: transaction spends a nonexisting siacoin output
2016/08/07 23:10:30.593817 storageobligations.go:396: Error with transaction set, redacting obligation, id 3d7ab5343b8b67483a26bd954e165e5e2c0fee5f62bb69352b75c0a63c7b06fc

(repeats every 15s for ~30m)

2016/08/08 13:54:11.831661 errors.go:156: communication error: error with 172.17.0.1:49528: revision iteration failed: rejected proposed modifications: unable to verify revision: rejected for high paying renter valid output

(repeats every few minutes)

Any idea what might be going wrong?

Full log: https://gist.github.com/mtlynch/4f75f75e74738d546ac7188913360b43

[DavidVorick]

A lot of the things that have been causing this have been addressed. You will still get large numbers of FormContract calls that fail, even with the new version. But largely, this is because people using the old version are talking to your host.

There were at least 3 underlying bugs that we fixed related to the problems reported here. If problems persist, they definitely deserve their own issue.