Whitelist and cleanup

[1credit]

Package Versions

Replace the values below using the output from npm list storj. Use npm list -g storj if installed globally.

StorjShare:  5.0.2
Core:          2.2.0
Protocol:     0.7.1

Replace the values below using the output from node --version.

v4.4.6

Expected Behavior

Only contracts and shards from legitimate bridges should be accepted.

Actual Behavior

Today, anyone can create a bridge server an inject junk into the network. Doing so will consume space without compensation.

A whitelist of known good bridges needs to be included, ideally as an external file for easy updating, but hardcoded would work for now.

Once provided, a command must be created to scan farmer.db directories and remove unpaid contracts/shards.

This issue is based on the assumption that every bridge owner is responsible for making their own payments, and that naturally, some will not.

Steps to Reproduce

Please include the steps the reproduce the issue, numbered below. Include as much detail as possible.

1. ...
2. ...
3. ...

[super3]

So the core issue here is that some bridges are not paying for contracts. If these bridges share data with us, we can actually pay for them out of Test Group C.

[littleskunk]

As farmer we are not able to force them to share data. -> We need a whitelist to ignore all unpaid bridge server.

[vedalken254]

I would say do it as a hardcoded URL personally but that's me.

[1credit]

Mixed emotions on hardcoding. Feels more like something that should be in the users control, unless, of course, its a vector for attack (where anyone allowing a non-whitelisted bridge in exposes everyone to unpaid work). However, if hardcoded, and someone wanted to attack, they would just pull the open source code down and change the list. If that is all it took, it would be worthless either way.

Basically a mechanism needs to be put in place where honest players can participate without fear of loading junk onto their systems - ideally by simply running code they install themselves from an authoritative location such as github. That code needs to have the whitelisting filters in it being requested.

[vedalken254]

@1credit that's why i suggested a URL. Hardcode an authoritative URL containing whitelisted bridges into the client and have an option to disable authoritative requirements. Though the issue then becomes a reputational one. I'd much rather have bridges provide verifiable reputation data and give the client a setting where they define the minimum acceptable reputation value to accept contracts from.

[BlackDuck888]

@super3: maybe it will be helpfull if someone post a manual which data have to shared with you, and how to upload this data.

the current procedure is simple crap. every farmer starts with the announcement, and after a month the rules got redefined. if you want to kill this project, yes you are on the right way.

[littleskunk]

I am not sure a URL is a good solution. My NAS is to slow for leveldb. I would add only api.storj.io to my whitelist and ignore storj.dk as well. That should allow me a higher income.

Farmer with more CPU power can go for storj.dk contracts and add them to there whitelist.

A centralized whitelist is not possible in this case.

[littleskunk]

My hardcoded whitelist: https://github.com/littleskunk/core/tree/white-list

If somebody could add a config for this we can open a pull request and merge it.

[vedalken254]

Could we do it in a similar fashion to config.json for CLI? And ultimately, we'd want a team-hosted solution to deploy configuration updates for bridges that share data with the team/are controlled by the team.

[vedalken254]

Could do it as an update check to github even for that file.

[super3]

I think whitelisting bridges is a bad idea because usage is so low right now. These bridges might pay in the future, and you are stopping them before they even get out of the gate.

[BlackDuck888]

@super3 The design/structure will defined by the team with the rules/rewards.

right now it is more profitable to "farm" on storj.io directly, if this change i will change. every day which it take will make it harder to reverse.

[super3]

@BlackDuck888 That is incorrect. We actually have paid out 10x more for bridges other than our own, because they stored 10x more data at the current time.

[1credit]

Also, being whitelisted is a GOOD thing, if I bridge is NOT on the white list, communication is not allowed.

[littleskunk]

@super3 I understand that point. For the storj network other bridges are important. It should be a p2p network and not a centralized network with only one bridge server.

On the other side storj is performing an attack (https://github.com/Storj/core/issues/154#issuecomment-238121234) and the only solution for farmer is to ignore all unpaid shards.

My hardcoded whitelist will ignore storj.dk as well. We should change that later. I have to do it because the shard distribution is based on a low ping. I will get storj.dk shards only if I don't block them. See https://github.com/Storj/core/issues/359

@1credit The whitelist will not block the communication. You will relay PUBLISH messages. The only difference is that you will never send any OFFER messages to them.

[vedalken254]

The reason i suggested whitelisting is because we need data to be shared. Otherwise, you're not actually looking out for the farmers and instead looking for what makes the network seem bigger. Without farmers, this project can't go anywhere. Same with bridges but we need a compromise somewhere.

[BlackDuck888]

@super3 Currently Storj is an unpredictable blackbox if i look to the reward system.

As farmer i don't know for what i get rewarded, for used space, for traffic, how is this weighted?! I can't see if every thing is working correct for me. Is my hole space recognized, is all traffic counted? There are no stats at all. i can't compare to other farmers, i can't dig into to search for problems without any information.

if there is a hint that farming only storj.io is more profitable, i will use that. This work like mining alt coins, is there a coin underrated more miners got involved in this coin. The miners/farmer following the money, nothing else, capitalism in its pures form.

i like to support storj as open souce project, but money is at first position. My systems consume electricity, traffic and wear disks, and i put in time for test, setup, update or debug.

We need more transparency for the rewards! The Rules must announced befor a round start. We need detailed stats per farmer.

I did some mass uploads last days, the distribution of shards are not random, there are some host with a great advantage, they are reapeting in my upload logs while others are newer shown.

[super3]

Telemetry report system by itself generated millions of transactions. The detailed stats system you want is already on the roadmap but will take hundreds of man hours to build and display. Unless you want to submit a pull request it's going to take time to build the tools to automate that process.

As we said we pay flat rate for the bandwidth, storage, and we also pay for telemetry reports.

On Aug 22, 2016 3:05 AM, "Marc Schubert" notifications@github.com wrote:

@super3 https://github.com/super3 Currently Storj is an unpredictable blackbox if i look to the reward system.

As farmer i don't know for what i get rewarded, for used space, for traffic, how is this weighted?! I can't see if every thing is working correct for me. Is my hole space recognized, is all traffic counted? There are no stats at all. i can't compare to other farmers, i can't dig into to search for problems without any information.

if there is a hint that farming only storj.io is more profitable, i will use that. This work like mining alt coins, is there a coin underrated more miners got involved in this coin. The miners/farmer following the money, nothing else, capitalism in its pures form.

i like to support storj as open souce project, but money is at first position. My systems consume electricity, traffic and wear disks, and i put in time for test, setup, update or debug.

We need more transparency for the rewards! The Rules must announced befor a round start. We need detailed stats per farmer.

I did some mass uploads last days, the distribution of shards are not random, there are some host with a great advantage, they are reapeting in my upload logs while others are newer shown.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Storj/core/issues/327#issuecomment-241330129, or mute the thread https://github.com/notifications/unsubscribe-auth/AADuL83K4FffqD6Zqj9csG21KPzc52uoks5qiUpAgaJpZM4Jj9EA .

[tacticalchihuahua]

@BlackDuck888 you want both "capitalism in it's purest form" and a "fair rewards system"?

Couldn't resist. 😄

[BlackDuck888]

Ok, you got me. First i want to know how the rewards work right now. Then i am able to optimize my setup.

oh one of my commets are gone :(

[tacticalchihuahua]

It's pretty straightforward. We pay a rate for data stored (lower), a rate for data transferred (higher), and a flat amount per telemetry report. @frdwrd can provide those numbers.

[BlackDuck888]

the flat amount per telemetry report means, payed for sending the telemetrie report, a host stored 1TB will get the same as someone storj 2TB, if both sending reports over the same time? this is helpfull for me.

what is transferred data? Up and download? Or only for Downloads? Only for the first famer or for all redundant data farmer?

What are rates for storing, transfer and reports and what are the units?

[1credit]

@blackDuck888 Sorry, but this is WAY off topic for this github issue. Could you please create your own issue, or discuss on one of the forums?

[littleskunk]

We already have an issue for this: https://github.com/Storj/bridge/issues/178

[1credit]

Thunks skunk. Can you clean this issue up to keep it focused?

[littleskunk]

Available with the next storjshare-cli version

"network": {
  "renterWhitelist": [
    "<renterNodeID>"
  ],
},

Logfile will look like this:

{"level":"debug","message":"pending offers 0 is less than concurrency true: 3","timestamp":"2016-09-22T19:37:54.065Z"}
{"level":"debug","message":"renter is whitelisted: true","timestamp":"2016-09-22T19:37:54.065Z"}