Start a new chain from genesis block for NEO 3.0

[erikzhang]

I'm considering whether to start a new chain from genesis block for NEO3.0. Here are some ideas for your reference and discussion.

Why do we need to start a new chain?

Here are several reasons.

1. Stale code

   We created NEO from the code of Antshares, with a large number of deprecated features that are contained in current code and block chain. This makes it difficult for new developers to understand NEO's code, and new tools have to parse the data generated by features that are no longer supported. If we give up the old data, we can have a cleaner start.

2. Difficult to make architectural adjustments

   The community has made some improvement proposals that cannot be implemented without making huge structural adjustments. One of the more important proposals is to transform neo and gas into NEP-5 assets and to abolish the concept of global assets. I will explain these proposals in detail in the following paragraphs.

3. To decouple NEO and NeoVM

   In order to decouple NEO and NeoVM, some of the instructions in NeoVM need to be removed, which will cause the old contracts to fail, and it would be best if it could be done together in NEO 3.0.

What new features can we implement in NEO 3.0?

1. Abolition of UTXO and global assets

   In NEO 2.x, we have two different ways to create assets, the first is to create a global asset based on UTXO, and the other is to create a NEP-5 token that is stored in a contract. For a long time, the biggest challenge in the development of smart contracts on NEO is to handle global assets in contracts.

   Why don't we put the global assets into the smart contract in NEO 3.0? As a result, the development of smart contracts will become very easy. The only problem is that it is difficult to transform neo and gas into NEP-5 tokens under the existing architecture.

2. Unique type of transactions

   In NEO 2.x, we have many different types of transactions:

   • MinerTransaction
   • IssueTransaction
   • ClaimTransaction
   • EnrollmentTransaction
   • RegisterTransaction
   • ContractTransaction
   • StateTransaction
   • PublishTransaction
   • InvocationTransaction

   Most of them are obsolete, and the remaining types of transactions can be easily implemented in smart contracts. The only thing we need to keep is InvocationTransaction.

3. Redesign the instruction set of NeoVM

   Currently, the contract invocations are done through the APPCALL instruction (and the corresponding stack isolation instructions), and NeoVM will directly look for the target contract. This is very unreasonable for NeoVM, because the list of smart contracts on the blockchain should not be directly visible to NeoVM.

   We need to create a new API to invoke contracts in a SYSCALL and discard the original APPCALL instructions.

   There are other instructions that increase coupling, such as CHECKSIG and CHECKMULTISIG. They are widely used in the current system.

What about NEO 2.x?

When the NEO 3.0 mainnet is started, the 2.x network will be stopped and the data is retained, but no new blocks are generated. The data/assets on the 2.x network need to be migrated to 3.0 network.

The affected populations are:

1. neo or gas holders

   The NEO Foundation will distribute new neo and gas to the holders on the NEO 3.0 network, these new neo and gas are based on NEP-5 standard. The users do not need to do anything. Historical transactions are not migrated to the 3.0 network, but users can query them on 2.x network at any time.

2. Other tokens/assets holders

   For those who hold global assets or NEP-5 tokens, they need to swap or accept airdrops on the NEO 3.0 network with the help of the project party.

3. Global assets

   If you've ever registered a global asset on the NEO 2.x network, the NEO Foundation will assist you in migrating it to the NEO 3.0 network and registering a new NEP-5 token for free.

4. Smart contracts

   If you've ever created a smart contract on the NEO 2.x network, the NEO Foundation will help you redeploy it to the 3.0 network for free. But you need to migrate the data manually.

5. Exchanges

   Exchanges need to swap tokens for the users, including neo, gas and other tokens.

When will this happen?

NEO 3.0 development is a long-term process that takes at least 1-2 years to develop and test. Once the NEO 3.0 mainnet is ready to start, we will announce at least 2 months in advance and provide all the help needed for the migration.

During NEO 3.0 development, any DApp development should be based on NEO 2.x and no compatibility issues need to be considered.

Where to discuss?

If you have any thoughts on NEO 3.0, please discuss it below this post.

If you have any good ideas for the development of NEO 3.0, you are welcome to send your proposals at https://github.com/neo-project/neo/issues. I will organize and label the proposals related to NEO 3.0 for discussion.

If you want to track the progress of NEO 3.0, you can visit our milestone on GitHub.

[EdgeDLT]

I can't speak from a technical perspective (will have to wait for the developers to explain any issues that may arise), but on paper I support the concept. As long as all users receive precisely the tokens they had on 2.x, and all existing projects can migrate their contracts for free, I think it's fine if the technical benefits are worth it.

EDIT: One thing I'd like to know is what happens to NEO 2.0 improvements during NEO 3.0 development? If we leave NEO in the current state for 1-2 years, we may be discouraging developers from launching on NEO until after 3.0 and falling behind our competitors.

[erikzhang]

One thing I'd like to know is what happens to NEO 2.0 improvements during NEO 3.0 development? If we leave NEO in the current state for 1-2 years, we may be discouraging developers from launching on NEO until after 3.0 and falling behind our competitors.

Don't worry about this, those new features that are compatible with 2.x will be developed first and merged into the 2.0 version.

[colinclosser]

I show 100% support for this long-term vision.

[xStarBlade]

Anything that solidifies Neo's position as a top tier public blockchain for the future should be implemented in my opinion. There might be some growing pains, but in the long run it's worth it if current Neo/NEP-5 holders see an appreciation of their token because of an excellent upgrade to the technology of NEO. As long as people continue to retain the value of their Neo assets in the transition, there is nothing to really complain about other than going through the effort of swapping coins out. But crypto owners should be well aware that holding your own money comes with responsibility by now.

[EdgeDLT]

@dhghabhjg This is quite inaccurate.

NEO 3.0 is a work in progress with many different areas to it, most of which have been experiencing development already. As Erik has mentioned, improvements that are compatible with NEO 2.0 will also be merged into NEO 2.0 first, so we will continue to see NEO improvements while 3.0 is pursued.

As for decentralization, I think we'd all like to see that happen quicker, but this is a fundamentally different approach to consensus and we should absolutely be taking our time to ensure we don't fuck it up.

The NEO 3.0 update does not affect decentralization; that's all about getting new consensus nodes online (architectural decentralization) and allowing the token holder to take over voting (political decentralization). This will happen over time regardless of any changes to the core platform.

[igormcoelho]

Let's go for the new NEO :D you can count on us!

[kevaundray]

What should those who are building the nodes do? If the nodes are built for 2.x , if will be redunadant code in 1-2 years.

Can the changes be done with a series of hard forks?

Edit:

Can you provide information regarding the consensus protocol; are there plans to change it?

[deanpress]

NEO 3.0 development is a long-term process that takes at least 1-2 years to develop and test.

[...] new features that are compatible with 2.x will be developed first and merged into the 2.0 version.

With NEO 3.0 in the spotlight as "NEO's big update", instead of these individual updates being perceived as positive, rapid development, they will be perceived by many as "part of the road to NEO 3.0, which will take a very long time". That's a negative underlying premise.

Companies that are looking for a suitable public blockchain will not be favorable towards a platform where this is the general premise for the next 12-24 months.

But this doesn't have to be the case.

My proposal is that these new updates (which build towards what will eventually be NEO 3.0) are to be introduced as their own individual and respectable updates (think Ethereum Casper, EOS Dawn 1.0, 2.0 etc), so as to eliminate the impression that they are merely stepping stones towards NEO 3.0.

This way the individual updates could be received in a more positive manner by the community and potential new projects, without unnecessary concerns for NEO 3.0.

Tl;dr: Take the one big spotlight away from NEO 3.0, and shine it upon individual updates instead.

NEO 3.0 will come when it comes, and should not be a concern or talking point with every new update for 2 years to come.

[anthdm]

First of all, I like most of the ideas that are proposed in NEO 3.0, so don't get me wrong here.

I think before creating a new blockchain--cause that's what NEO 3.0 actually is-- we need to set some things right. (1) for the people that believed in this project in the first place. (2) for the people that actually deployed smart contracts and building their businesses on the NEO blockchain.

1. Adding more network instrumentation/metrics will let us monitor the network more in-depth, this to optimize the p2p network and its known issues. Hence also reflects on the current consensus implementation, with more in-depth instrumentation/metrics we can detect, improve and enhance the current consensus protocol much faster and more accurate. NEO doesn't need 100k TPS it needs a stable platform first. dBFT is not a bad consensus protocol it is just not very well executed/optimized enough today.

2. Fix all the known VM issues that can potentially cause security vulnerabilities ASAP.

3. Smart contract development on NEO is a pain in the ass. We need to incentive and allocate more resources to developers that will improve this. Think about a truffle framework. We don't need 10 smart contract languages, we only need 1 good implementation with full documentation. The former can be done after we have stabilized.

4. We have enough block explorers and monitors.

5. We (the community) need to help writing a global documented specification for NEO. It is still too hard to find the right documentation, especially for creating new implementations of the protocol. We need a loud and clear NEO whitepaper that new implementations can follow from start to end.

6. I don't care about decentralization for now but let people vote for consensus nodes. this is promised to the community a long time ago.

7. I'm pretty certain we can implement a lot of the new features that are proposed for NEO 3.0 without even to rebuild this from scratch. Let's start with getting the foundation of the architecture stable and build from there.

Peace brothers

[anthdm]

@deanpress has a good point!

If ETH has done what NEO is planning to do than ETH was not the ETH it is today and certainly not what it will become in the future. Nothing is perfect from the start, how temping rewriting things maybe, we have a community that counts on us.

[deanpress]

If you've ever created a smart contract on the NEO 2.x network, the NEO Foundation will help you redeploy it to the 3.0 network for free. But you need to migrate the data manually.

A fundamental aspect of blockchain is that data is permanent and immutable.

Manual smart contract/data migration raises 2 questions:

1. If data must be migrated manually by the owner, what will be the methods in place to verify that the owner hasn't manipulated any of the data prior to migration?
2. What if the owner of a popular closed-source smart contract has become unresponsive? Does this mean the smart contract cannot be migrated and can only be kept alive on a hard fork?

[vncoelho]

Hey, @erikzhang. Great news in such reasonable and honest words. I fell that it represents an important path for the future of the Neo Ecosystem. In particular, considering the point you mentioned about keep applying all compatible improvements during the journey. Congratulations to you and your team for this insight and direction.

PS: If some problem happen, do not worry, we can keep the 2.0 working and freeze the NGD tokens....aehauehauheauea :dancing_men: :dancer: Just kidding! Count with us.

[saltyskip]

I am also for these changes. UTXO model needs to be retired. The biggest concern I have about NEO 3.0 is resource allocation. It seems like the dev community is often running blind on what the best things to develop are, and their efforts seem wasted.

Clear, Concrete, projects and objectives are needed in the roadmap for 3.0.

Looking forward to the next phase of NEO's evolution 👍

[erikzhang]

A fundamental aspect of blockchain is that data is permanent and immutable.

The data is still permanent and immutable. In the process of migration, the old data will still exist on the NEO 2.x network, and the data will never be deleted. For NEO 2.x, it simply no longer generates new blocks, but the data does not disappear.

If data must be migrated manually by the owner, what will be the methods in place to verify that the owner hasn't manipulated any of the data prior to migration?

As I said earlier, the old data is still there. If the owner of the contract tampered with the data during the migration, it would be easy to spot.

[ThomasLobker]

If there will be a new genesis block then it would be great if existing contracts and their data would be migratable to the new chain. Not every contract is a NEP-5 token, we will see other storage contracts as well. Airdrops will not always be possible.

I'd hate to see NEO lose it's UTXO architecture. I still think UTXO is more robust than the current NEP-5 implementation.

New genesis block and effectively going forward without history will probably be a dealbreaker for some (new) projects. For example if contracts are working with time locks or doing things with the block height.

All together I can see some positive aspects about a fresh start, but it's also giving us a lot of uncertainty. It seems a bit like an easy way forward, but maybe not the best. I'm looking forward to reading more opinions about this.

[Sinepitiss]

If you've ever created a smart contract on the NEO 2.x network, the NEO Foundation will help you redeploy it to the 3.0 network for free. But you need to migrate the data manually.

Does that mean that once I try to redeploy my smart contract ,I will be able to change my smart contract logic?

[erikzhang]

I'd hate to see NEO lose it's UTXO architecture. I still think UTXO is more robust than the current NEP-5 implementation.

You can still implement UTXO in smart contracts.

[ThomasLobker]

You can still implement UTXO in smart contracts.

Yes of course, but I'm not sure if this would scale well. You would need to handle all transactions inside the NeoVM. If you have many inputs, you would need many invocation arguments. And I'm not even sure how to handle transactions which would contain inputs of multiple addresses that all need their own signature.

And to make it NEP-5 compliant we would need to have a wrapper, that means we would need to do all the work like picking the best UTXO inside NeoVM. Maybe it's better to have a separate token standard for UTXO tokens, not compliant with NEP-5.

[erikzhang]

I'm not saying neo will be a UTXO based contract asset in NEO 3.0. But I'm still happy to discuss its feasibility.

To make this scenario more scalable, we can use the Native Contract I proposed earlier. https://github.com/neo-project/neo/issues/303

In a native contract, we don't even have to worry about performance because we don't need to start a VM at all.

[deanpress]

As I said earlier, the old data is still there. If the owner of the contract tampered with the data during the migration, it would be easy to spot.

@erikzhang Is there any repercussion for data tampering during this process? I'm also interested in what would happen if a closed source smart contract was deployed on 2.x by an anonymous and unresponsive owner. How does that contract migrate to the NEO 3.0 chain along with all its data?

[Nikolaj-K]

@ThomasLobker Why do you like UTXO, really? @erikzhang Doesn't UTXO require you to run over lists where, apart from the max upper bound, you don't know when you'll be done? Sounds plenty complicated for a smart contract. How will the trigger of incoming NEO and Token situation look after the rewrite? Any ideas on merging the efforts with the validator voting system?

Regarding data, when we're already renewing stuff, do we maybe want more fields of info being associated with accounts (addresses) on-chain, like a key-value dicts that user can create and edit per invocation, and that can be queried from smart contracts? Such are needed and created from scratch in about nine out of ten contracts. One can have users stake Gas for temp storage or something if we want.

[ThomasLobker]

@Nikolaj-K I like it because it's tamper-proof by design. Just like blockchain is a chain of signed blocks, where you cannot remove any blocks without damaging the integrity of the chain, with UTXO you cannot remove transactions without breaking the integrity. UTXO is more strict than simply keeping track of balances. You never have to update records, you only have to add records, this is more in line with the immutable character of blockchain. Besides from a technical point of view it's way easier to verify balances, since the references are already there in any UTXO transaction. You can place it in a relational ACID compliant database server with foreign key constraints and have perfect finality. No issues with transaction ordering or potential double spending.

[Jim8y]

Will you add some storage verification mechanisms to nep5 in NEO3? Since in the NEO 2.x, even the chain is synchronized, the storage might still be different(transaction executing error due to different versions in different nodes), and no exception will be thrown.

[tanitall]

I truly appreciate the fact that the NEO team is being transparent with why and when they're considering to make these changes/improvements. It's shows maturity and enables open discussion as many above are doing.

[Ethour]

I couldn’t agree more with @deanpress. We can’t unveleid the Neo 3.0 (decentralization, voting system, economy model, neo-sharp, tps etc etc) at the beginning of July and after one month discuss about a new chain for Neo. I’m not a developer, I’m a fan of Neo project since Jan 2017. I saw the Neo growth day by day, I saw the first messages of CoZ and I saw the coin value touch 200$.

As a community member I think Neo has to improve its tech step by step. Thinking a one big jump of 24 months waiting the new chain, it could be a negative impact to our community and make no-sense every improvement to Neo 2.x or any new project/partnership in this “waiting period”.

Today there is an ama on reddit. Which is the link between the community’s questions and this discussion?

I apologyze for my input, take in consideration also the community sentiment. Go Neo!

[Jim8y]

NEO3 should be a revolution, not an optimized NEO2.x. I believe a lot of new advanced features are simply not compatible with NEO2.x and thus can't be transmitted to NEO2.x. NEO is still a rapidly developing project, give it more chance to explore.

[Ethour]

I absolutely want to give it “more chance to explore” but I can’t see our direction.

First, decentralization by the end of 2017 and NeoX & NeoID in first half of 2018.

Then:

• NeoID—>DNA—>Ontology (??)
• NGD road map (may 2018) https://neo.org/blog/details/4086 NeoX—> Test in Q4 2018
• Neo 3.0 (july 2018) https://neo.org/blog/details/4091

Now, new chain with new genesis block in 2 years. These different explorations create confusion to me.

[osmirnov]

In the ideal world, NEO delivers Neo 3.0 on Neo project and CoZ delivers NeoSharp totally compatible (Neo 2.0) and modular, covered with tests and ready to plug in all new features. It depends on time and resourcing where we would meet but maybe Neo 4.0 could be NeoSharp based. It addresses all concerns people raised about timeline if NGD does Neo 3.0 on its own and gives NGD plenty of time to make a shift without the community pressure but with CoZ international team benefits.

[mwherman2000]

What parts of NEO 3.0 are truly unique to the NEO platform? ...for example, the chosen consensus algorithm(s) ...and plug these directly into the existing highly-composable Stratis Platform C# blockchain framework?

public class Program
{
    public static async Task Main(string[] args)
    {
        try
        {
            var nodeSettings = new NodeSettings(protocolVersion:ProtocolVersion.ALT_PROTOCOL_VERSION, args:args);

            IFullNode node = new FullNodeBuilder()
                    .UseNodeSettings(nodeSettings)
                    .UseBlockStore()
                    .UsePosConsensus()
                    .UseMempool()
                    .UseWallet()
                    .AddPowPosMining()
                    .UseApi()        
                    .UseApps()
                    .AddRPC()
                    .Build();

            if (node != null)
                    await node.RunAsync();
        }
        catch (Exception ex)
        {
            Console.WriteLine("There was a problem initializing the node. Details: '{0}'", ex.Message);
        }
    }
}

Reference: https://github.com/stratisproject/StratisBitcoinFullNode/blob/master/src/Stratis.StratisD/Program.cs

@ashrolls

[Flankarooo]

Yeah let's do this :D But keep it between us.

[EdgeDLT]

In addition to free redeployment of smart contracts, we should probably consider some GAS/NEO contributions to any smart contract developers that will need to update their contracts to be compatible with changes.

We should also look for automatic data transfer solutions. As @deanpress mentioned on the Discord, smart contracts will not remain trustless with manual transfers (introduces an attack vector and the chance of data manipulation/loss during migration).

[lock9]

I think the core team has a consensus that this is going to be implemented. We will need a "large" genesis block to make sure all data is present.

Do you guys want to rename this issue or create a more "focused" one? We should care about data migration by the end of the year only. @neo-project/core opinion? We are doing this, right?

[shargon]

Allow genesis's transactions 0 gas fee and transactions size higher than the current maximum size

[igormcoelho]

There are many possibilities for the migration... we could pre-load migrated contract data before genesis block (so as migrated balances and migrated storage), and pre-start on "genesis" with that everything loaded, no need for actual transactions. I don't think much stuff is supposed to be migrated like this. After that, I believe that cross-chain mechanisms are supposed to allow specific asset migrations to occur, in a much larger and smoother period (years perhaps).

By the way, this is a very beautiful and heterogenous thread... we should keep it long time... open until Neo 3 launches ;)

[lock9]

@igormcoelho but can we have other smaller, "achievable" tasks (issues)? I think the way this is proposed looks more like a milestone. Nobody is going to send a single PR for this. I think someone should propose a design first. I will create an issue for that. This should be released by Q4 only

[shargon]

The no-limit-transaction is better in the block 1, because the block 0 is on memory every time

[lock9]

@neo-project/core This looks done, shouldn't we close this? Because I can't see what we are waiting for, can anyone clarify? If there are other tasks to be done, can't we create a new issue? Thanks

[cooncesean]

Hey guys, wondering if the development team has any guidance (for wallet developers) as to when v3.0 is slated to be released (more importantly, when we should target upgrading our node + wallet software).

Apologies if this is the wrong forum to ask ... I am failing to make any headway in Telegram + Discord.