search

near / bounties

Specs for technical and non-technical work that earns NEAR tokens
https://devgovgigs.near.social
72 stars 8 forks source link

[500 DAI] Implement a time machine for NEAR testing infrastructure (rust, python) #31

Closed SkidanovAlex closed 3 years ago

SkidanovAlex commented 4 years ago

NEAR Bounty Terms

Before beginning work on the bounty, you must submit a proposal. Only if your proposal is accepted will you be able to claim the reward of the bounty.

There should be at most 1 week between proposal is accepted and you start working on the bounty. It should take you at most 2 weeks to complete the bounty.

Description

NEAR tests are written in python and use a rich library that allows cluster manipulation, network interference, communication with nodes, issuing transactions and others.

NEAR reference implementation nearcore has a compile-time flag (so-called "adversarial" flag) that enables certain features (so-called "adversarial" features) that are disabled in the golden bits, but allow deeper-level testing when enabled.

In this bounty we ask you

  1. To implement necessary "adversarial" feature and the python library functions that would allow controlling the passage of time in the nearcore, such that all the places where Utc::now() is used today still use Utc::now() without the adversarial flag, but allow explicitly stopping and interacting with time via adversarial features when the flag is enabled.

  2. Add example test that use these features: the test should use the end-point to stop the time, and then make sure that in the next 5 seconds no approvals / blocks / chunks are sent over the network (we have infrastructure that allows examining all the messages passed over the network). The test should then jump the time 10 minutes forward, and observe (by monitoring the network) that the block producers send approvals for at least 50 consecutive target heights.

Context

  1. The existing infrastructure is explained here: https://github.com/nearprotocol/nearcore/wiki/Writing-integration-tests-for-nearcore.
  2. Nearcore repo is here: https://github.com/nearprotocol/nearcore, search through the occurrences of the "adversarial" flag for examples of how it is used today.
  3. chain/chain/src/doomslug.rs is an example of a module that already uses time that is passed to it from outside. Many other places use Utc::now(). We want to replace places that make decisions or have side-effects (such as the occurrence in Block::produce()), but not those that are used to periodically invoke some logic (such as those related to usage of ClientActor::block_production_next_attempt). We are OK to keep Utc::now in the existing rust-tests.

Acceptance Criteria

The work is accepted if all the following work items are completed.

Ideally perform the following workitems in order, and confirm with us the correctness of each one before moving on to the next one:

Bounty

500 DAI

gitcoinbot commented 4 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 500.0 DAI (500.0 USD @ $1.0/DAI) attached to it.

gitcoinbot commented 4 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Workers have applied to start work.

These users each claimed they can complete the work by 265 years, 6 months from now. Please review their action plans below:

1) pczarn has applied to start work _(Funders only: approve worker | reject worker)_.

Hello. I have decent experience with Python, Rust and integration testing (used Rust for the last 7 years).

The task's description is clear. I have yet to figure out where the RPC server resides and how it works. I'll get back to you with questions if any arise.

Later, I would be open to getting involved with other tasks in the NEAR project. Best wishes. Peter 2) tota999 has applied to start work _(Funders only: approve worker | reject worker)_.

0x9aD952a23d9b85bcA0a6D102E28A8ebb39707DB8 3) brodkes has applied to start work _(Funders only: approve worker | reject worker)_.

tools and materials for making

Learn more on the Gitcoin Issue Details page.

gitcoinbot commented 4 years ago

@vporton Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

vporton commented 4 years ago

@vporton Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

I am going to start work today or next day.

vporton commented 4 years ago

https://github.com/near/nearcore/blob/774ac1fe95ef6d1e1741517b7bb46f989c767c32/chain/network/src/routing.rs uses Instant::now(). Should them also be replaced (like Utc::now())?

vporton commented 4 years ago

I realized that my Rust skills are yet not enough (I should keep learning Rust programming). Sorry for the time lost.

I do not see "Stop work" button at https://gitcoin.co/issue/near/bounties/31/100023777 - can anyone explain how to refuse to fulfill this task?

pczarn commented 4 years ago

Hello @SkidanovAlex. I applied for the task. I heard the @gitcoinbot may be malfunctioning. It didn't post here 2 days ago when I applied. I'm not sure whether you got notified about me applying through gitcoin itself.

pczarn commented 3 years ago

I'm down, being sick. I will be back within a few days.

cameron-NEAR commented 3 years ago

Hi @pczarn, how is the completion of this bounty going? I'm here to help if you have any questions.

pczarn commented 3 years ago

Hi @cameron-NEAR I'm almost done and will submit for second review soon. The progress on this one is most likely 99% beause one test is failing (sync::test::test_slow_header_sync), and the rest is done.

pczarn commented 3 years ago

This is a question best directed to @SkidanovAlex but you may take a look :)

@SkidanovAlex :

I'm changing the test infra to start proxy processes after the nodes are started, serially. Without that Popen sometimes hangs if the proxies for some processes happen to start before some other processes.

How can I reproduce the problem with Popen?

I noticed that proxy-related tests such as sanity/proxy_{simple,example} are failing. After some digging, I found that the mentioned change is the cause (https://github.com/near/nearcore/pull/3302). It appears that proxies do not spin up fast enough to accept inter-node connections before they are initiated. I will be looking for solutions to the issue.

I need proxies to be functional as a part of near/bounties#31 and #3661

pczarn commented 3 years ago

Please review my code. I already reviewed it to spot obvious mistakes.

The proxy startup problem is yet to be resolved or investigated

SkidanovAlex commented 3 years ago

@pczarn I will not be able to review today, but will be sure to allocate time next week. Do proxy tests also fail on your machine before your change, or only after?

pczarn commented 3 years ago

@SkidanovAlex Ok thanks.

The tests fail only before my change to cluster.py. Never after. Tried this on my machine only.

pczarn commented 3 years ago

@SkidanovAlex please allocate some time for a review

cameron-NEAR commented 3 years ago

Hey @pczarn, @SkidanovAlex reviewed your bounty submission and said it looks great!! Can you message me on Telegram at @camerondennis to complete the payment?