search

CartoDB / cartodb

Location Intelligence & Data Visualization tool
http://carto.com
BSD 3-Clause "New" or "Revised" License
2.75k stars 652 forks source link

Test parallelization pending specs #5906

Closed jesusvazquez closed 7 years ago

jesusvazquez commented 8 years ago

I've been working in test parallelization for a while. Right now we can actually start a parallel execution running the specs in batches with gnu parallel, specifying the batch size with the -j (jobs/threads) parameter.

The batch size is limited by the machine processor in terms of performance. I'm using a 4 core droplet running the specs in 12 threads batch size and it takes 10-11 minutes to run all the specs. That being said, running this with a better machine with more cores means less time.

First workflow was defining the batch size, for example 12 and then creating 12 databases and starting 12 redis but some specs failed randomly because they were using non empty databases. I had to find some deterministic results so I had to change the workflow.

Right now I'm generating the speclist by performing some cat,greps and seds to the Makefile and after that I count the lines. I create as much databases as specs we have to resolve the problem I had before. Then I define the batch size, for example 12 and I start 12 redis servers in different ports.

I've been testing this with multiple executions and the results are now more deterministic. By now we have this errors:

// DEPRECATED

Random errors

//

cc/ @luisbosque @Kartones @juanignaciosl @ethervoid

Kartones commented 8 years ago

Assign to me at the second and third ones (vis collection and layers controller), but I'll need the branch and how to run them to reprouce the error, but for sure I'll stabilize those.

Data imports one will need some changes in how it fakely uploads files, but if @rafatower team is too busy assign to me too, as this is important to start saving precious time running tests

Kartones commented 8 years ago

/cc @xavijam just so that he knows

jesusvazquez commented 8 years ago

The branch is named test-improvements and it actually has a few scripts in the root to make it easier but they will be placed somewhere else in the future.

In order to run this locally you have to install gnu parallel in your machine. I don't know if its actually in the ubuntu repositories but you can do it easily with the following steps:

wget http://ftp.gnu.org/gnu/parallel/parallel-20130622.tar.bz2
tar -xvf parallel-20130622.tar.bz2 
cd parallel-20130622/
sudo ./configure && sudo make && sudo make install

And test it with this command `parallel echo ::: A B C``that should return 

ubuntu@test01:~$ parallel echo ::: A B C
A
B
C

After this you only need to checkout the branch. To run all of them in parallel you need to generate the specfull.txt with the following command

ubuntu@test07:~$ sh generateSpecFull.sh 
# Speclist has been created

Then you need to start the redis and create the databases with the following command.

ubuntu@test07:~/www/production.cartodb.com/rsync$ sh wrapper.sh 20
# Wrapper finished

To start running all the tests execute this:

date; time parallel -j 18 --delay 0.1 -a specfull.txt './executor.sh'; date

If you want to run just one spec specifying the database and the redis you can run this:

PARALLEL=true RAILS_ENV=test RAILS_DATABASE_FILE=database_6003.yml REDIS_PORT=6003 bundle exec rspec spec/requests/carto/api/layers_controller_spec.rb

If you have any problem tell me as soon as possible to save you some time.

ethervoid commented 8 years ago

We should start using this http://apidock.com/rails/ActionController/TestProcess/fixture_file_upload to fake the file uploads using fixtures

jesusvazquez commented 8 years ago

/cc adding @gfiorav because he wrote spec/integrations/common_data_integration.rb we will check this error on tuesday.

jesusvazquez commented 8 years ago

Added two new conflictive specs.

juanignaciosl commented 8 years ago

Please let me know if I can help with this.

rafatower commented 8 years ago

is this ticket complete?

gfiorav commented 8 years ago

@jesusvazquez was testing it thoroughly. I think no problems so far.

jesusvazquez commented 8 years ago

The asset spec seems to be OK but sometimes it remains IDLE and never ends. This is a random issue.

If this happens you can either kill the job or kill the process inside the testing server and it will finish. Both are manual and not ideal "solutions".

I created a pull request (#6129) in order to deactivate the spec in the parallel execution and run it later in serial with the other failed specs. Again this is a temporal solution until we dig in and fix the issue.

The tests that have the same condition are the following:

rafatower commented 8 years ago

do you have any more clue about why they fail? a lock in db maybe?

the user mover ones strike as important to me.

jesusvazquez commented 8 years ago

It does require futher research in the logs for each execution. I'll keep this issue updated and will post more details for each spec.

Maybe the user spec is a good start but I want to clarify again that it is running okey sequentially.

alonsogarciapablo commented 7 years ago

There has been no activity on this issue for more several months. We are closing it. If you think this still needs to be addressed please open a new issue.