search

verilator / verilator

Verilator open-source SystemVerilog simulator and lint system
https://verilator.org
GNU Lesser General Public License v3.0
2.53k stars 606 forks source link

Make verilator_ext_tests head-to-head #1483

Closed veripoolbot closed 5 years ago

veripoolbot commented 5 years ago

Author Name: Todd Strader (@toddstrader) Original Redmine Issue: 1483 from https://www.veripool.org

Original Assignee: Todd Strader (@toddstrader)

This is still a WIP, but I think it warrants some discussion before I trudge ahead much further: https://github.com/toddstrader/verilator_ext_tests/tree/head-to-head

I'm proposing that we have Travis always pull the latest master from every submodule (including Verilator, which I added). This way we don't have to keep bumping these submodules by hand. Right now, I'm just running Travis on my branch manually, or by pushing verilator_ext_tests. If we go with head-to-head, we can initially just turn on daily cron builds.

There's also some fanciness in Travis about public/private keys, encrypted YAML and protected environment variables. This supposedly allows you to "safely" kick off Travis builds via webhooks. I'm wondering if we could grant these projects we're building with these privileges so that they could kick off a verilator_ext_tests build when they run their own CI. This would just grant them the ability to control Travis runs and not permissions to GitHub. All of this is to say, I would need to do more research on this bit.

You'll notice that Travis is currently failing the SweRV test: https://travis-ci.com/toddstrader/verilator_ext_tests/builds/120918474

This is actually a fortunate coincidence because the SweRV test passes if you run with the Verilator rev that is committed via the submodule (just don't update with --remote). But if you do update to HEAD of verilator, it breaks. So . . . heads up SweRV people (I'll post there).

I also learned how to cache the full Verilator build with Travis build stages. This has potential benefits for the Verilator proper CI. I'll have to circle back there. However, the solution I've landed on is a bit cumbersome. Essentially, I'd like to install Verilator with installation option #4 (from README.pod). However, this involves unsetting VERILATOR_ROOT which breaks driver.pl which these tests depend on. I'm wondering if VERILATOR_ROOT can be deprecated, but that's a topic for another thread.

What are your thoughts on this approach? It will obviously break more often. But I think that's what we want. If we have to manually bump submodule versions, I'm not sure how much more beneficial verilator_ext_tests is than these projects' own CI environments.

veripoolbot commented 5 years ago

Original Redmine Comment Author Name: Wilson Snyder (@wsnyder) Original Date: 2019-07-26T22:59:39Z

I agree this is a good approach. Note Verilator, along with SwerRV is part of Chips Alliance, so if you want you can ask for commit privileges to it, and/or we can move the ext tests to the CHIPS Alliance repo if that helps in some way (e.g. they might pay for more Travis services).

veripoolbot commented 5 years ago

Original Redmine Comment Author Name: Wilson Snyder (@wsnyder) Original Date: 2019-08-04T02:16:36Z

 captures the failures due to new WIDTH warnings.

veripoolbot commented 5 years ago

Original Redmine Comment Author Name: Todd Strader (@toddstrader) Original Date: 2019-08-22T20:13:44Z

There's more to do here, but I'm focused on the DPI protected modules for now. But I believe this should be land-able: https://github.com/toddstrader/verilator_ext_tests/tree/head-to-head

If we turn on daily cron jobs, we'll get a head-to-head check of all these tests and Verilator every day.

Also, the build has gone red since I looked at this last: https://travis-ci.com/toddstrader/verilator_ext_tests/builds/122567524

This looks to be a legitimate SweRV issue this time. I suggest we add a requirement to the README that external projects have their own CI using whatever their fixed version of Verilator is. We can grandfather in the current two projects and politely ask them to set up their own CI.

veripoolbot commented 5 years ago

Original Redmine Comment Author Name: Wilson Snyder (@wsnyder) Original Date: 2019-08-27T22:59:20Z

These changes look fine.

Personally I'd like to be able to run the tests pointing at a separate verilator tree (I am developing at that moment) without picking up the external for verilator. Do you know if there's a way we can do that? Or maybe I just live with the external fetch and set VERILATOR_ROOT to my local area?

veripoolbot commented 5 years ago

Original Redmine Comment Author Name: Todd Strader (@toddstrader) Original Date: 2019-08-28T09:59:09Z

That's what I've been doing. I just added ci/update_submodules.sh which does the remote submodule update for all submodules but excludes the verilator submodule if $VERILATOR_ROOT is set to something other than the in-tree copy. Does this work for your workflow?

veripoolbot commented 5 years ago

Original Redmine Comment Author Name: Wilson Snyder (@wsnyder) Original Date: 2019-08-28T11:54:27Z

Ah, driver error. Feel free to squash and push. If you didn't already please contact the SweRV team to report their bug.

veripoolbot commented 5 years ago

Original Redmine Comment Author Name: Todd Strader (@toddstrader) Original Date: 2019-08-28T12:39:23Z

Done and done.