New proj command and various infrastructure updates

[ginty]

Sorry about the size of this, another update that should have been fairly self-contained but which ended up requiring a lot of tangential infrastructure...

This adds a new global Origen command called origen proj which is for creating workspaces/views of a project comprised of multiple packages which can each be pulled from Git, DesignSync, a symlink, or a tarball and potentially other revision control tools in future.

Potential applications include things like building a test program directory from a collection of sub-blocks. The features of the command are likely to continue to evolve for a while as driven by use cases.

I held off documenting it until the doc system is available, but I'll demo at the core team this week and in a nutshell:

origen proj init   # Initialize a new project area with a bill of materials (BOM) file
origen proj create my_workspace   # Create a new workspace/view of the project BOM
cd my_workspace
origen proj update # Update the workspace per the latest BOM
origen proj bom    # View the workspace BOM which is the combination of the wsp and project BOMs

To enable this command quite a lot of basic infrastructure has been added and/or improved:

Macros

• I finally understood how macro availability works - within the crate that owns the macros there is also a file load-order component to it. To simplify, I moved most of the current macros to a new module origen::macros and loaded that first. Everything in there is available everywhere in origen, cli and pyapi.
• Everyone feel free to move any other macros there in the future, but for now I've only moved the ones that I have used myself.
• Added some macros to exit with a big SUCCESS, PASS, FAIL or ERROR - exit_success!(), exit_pass!(), exit_fail!() and exit_error!()
• Added an error! macro to generate an origen::Error, typical use example:

fn some_func(x: i32) -> origen::Result<()> {
  if x < 10 {
    // Equivalent code
    Err(origen::Error::new(&format!("Something is wrong, x is {}", x)))
  } else {
    // Note that this doesn't generate a 'return', that's still required where applicable
    error!("Something is wrong, x is {}", x)
  }
}

Logger

• Added a verbosity option to all current and future origen commands, use -v, -vv and -vvv to make the console output more verbose.
• Added a new log level log_trace for really low level stuff, think println!("I am here") type statements for debugging. This level is currently unique in that it doesn't appear in the logfile by default and only when the highest verbosity is enabled. All other log items always appear in the log file regardless of the console log level.
• Added macros that work like println!() for all log levels and also a macro called displayln!() which is just like println!() but which will also write to the log in addition to the console. See here for documentation on the log macros - https://github.com/Origen-SDK/o2/blob/0839b160bcf0ee2c5ef85e774069665aa66da138/rust/origen/src/utility/logger.rs#L1-L54
• Additional log files can now be created as described here - https://github.com/Origen-SDK/o2/blob/0839b160bcf0ee2c5ef85e774069665aa66da138/rust/origen/src/utility/logger.rs#L418-L443
• New log files also have the option of being made thread-specific so that the log will be writing to the given file for that particular thread only.

User Model

• Added an initial user model. The model for the current user is instantiated as origen::USER.
• This has a method to get the user's ID from the system and their password. The password method will prompt the user to enter it and then cache it in memory for repeated calls in the same thread.
• The password method is thread safe in the sense that if multiple threads query the password at the same time then only the first thread to arrive will prompt the user and the rest will hold off rather than prompting them multiple times.
• I experimented with long term caching using the system keyrings but couldn't get it to work on RHEL6. Hopefully RHEL7 and/or Windows will yield more success. This feature is behind a new optional build feature - password-cache.

Revision Control API

• Added initial revision control drivers along the lines of O1, added Git and DesignSync for now.
• This defines a RevisionControlAPI trait that all drivers should support - https://github.com/Origen-SDK/o2/blob/548650f8ed8e7b774ccf072f6e935dac2a67d6e0/rust/origen/src/revision_control/mod.rs#L46
• Currently only initial pop/clone and checkout methods have been implemented.
• The DesignSync driver is pretty simple and just executes the relevant dssc command and captures and processes the output.
• The Git driver uses the Rust git2 crate. This provides Rust bindings for a C library called libgit2, this gets bundled into the compiled Rust output and basically means that the Origen CLI can execute Git commands without the user even needing to have Git installed - could be useful in things like Windows envs. It also provides APIs to do literally everything in Git. The downside is that its really complicated and the Git driver ended up being much more complicated than the dssc one. On the other hand, it teaches you a lot about Git and hardcore Rust libs.
• Note that there are no Python bindings for these APIs yet.

Utilities

• I made a top-level utility module and generally I think a lot of what is currently in origen::core could move into here in due course with origen:;core eventually becoming origen::model.

• There's quite a few useful helpers for working with files and dirs in origen::utility::file_utils - https://github.com/Origen-SDK/o2/blob/548650f8ed8e7b774ccf072f6e935dac2a67d6e0/rust/origen/src/utility/file_utils.rs

• utility::command_helpers contains functions for logging STDOUT and STDERR from child processes - https://github.com/Origen-SDK/o2/blob/548650f8ed8e7b774ccf072f6e935dac2a67d6e0/rust/origen/src/utility/command_helpers.rs

  • Here is an example of using them - https://github.com/Origen-SDK/o2/blob/548650f8ed8e7b774ccf072f6e935dac2a67d6e0/rust/origen/src/revision_control/designsync.rs#L42-L48

• I also moved logger into origen::utility since I was working on it.

General Points of Interest

• I used Rust templating for the first time, a crate called Tera, which is great, just like J2 and other templating engines. It should be very easy to create a new app command with this and I also think a global origen c command could be interesting and which uses TOML or other config files for the parameters.
• Here is an example of a template - https://github.com/Origen-SDK/o2/blob/548650f8ed8e7b774ccf072f6e935dac2a67d6e0/rust/origen/cli/src/commands/proj/templates/workspace_bom.toml
• And an example of how to render it - https://github.com/Origen-SDK/o2/blob/548650f8ed8e7b774ccf072f6e935dac2a67d6e0/rust/origen/cli/src/commands/proj/mod.rs#L105-L114
• Here is an example of how to render a static file embedded in the Rust binary, which turned out to be super-simple - https://github.com/Origen-SDK/o2/blob/548650f8ed8e7b774ccf072f6e935dac2a67d6e0/rust/origen/cli/src/commands/proj/mod.rs#L28

When writing this README I learned that if you type y when looking a file in Github it turns the url into a permalink, good for links in PRs like this!

I think there might be some conflict with #107 due to some of the file moves in here, but I'll take care of that once that is merged.

[priyavadan]

a lot of good work on this PR @ginty , I am reviewing the changes you have made , primarily to increase my understanding of rust.

[ginty]

@coreyeng, any objection to merging this?

I've built my prog gen branch on top-of this and would like to open a PR for that soon to handle it in manageable pieces rather than dumping a huge change at the end.

[coreyeng]

Ah, sorry. I didn't have a chance to look at it when it was first opened and its slipped my mind since.

Nope, no objection from me! Looks good!