search

redhataccess / ascii_binder

AsciiBinder is an AsciiDoc-based system for authoring and publishing closely related documentation sets from a single source.
https://asciibinder.net
MIT License
76 stars 39 forks source link

Proposal: Remote repo support in Topic Map #87

Closed nhr closed 4 years ago

nhr commented 7 years ago

[edited with feedback from @indigo423 and @Fryguy]

Background

Structurally, the _topic_map.yml file is a YAML stream (i.e. multiple records). The records are each either topic groups or individual topics. All are assumed to be present in the docs repo that AsciiBinder is processing.

Proposal

I would like to extend AsciiBinder to support the inclusion of remote AsciiBinder-compliant docs repos as build-time topic groups within a local repo being processed.

In other words, given two repos:

There would be a way within _topic_map.yml to use the contents of my_other_repo as a topic group in my_main_repo.

Invocation

In order for a user to specify that a remote repo should be included, they will create an entry in _topic_map.yml that is similar to that of a topic group:

Name: Remote Group Name
Dir: remote_group
Distros: foo,bar<1>
URL: <see below>
RemoteDistro: baz<2>
<1> Optional <2> Optional, but unlike the regular Distros key, this is not a list. The URL value can follow one of two patterns: 1. `http(s)://hostname(:port)/path/to/archive.[bz|tgz|zip]` for archive files or 2. `http(s)://hostname(:port)/path/to/repo(.git)(@)` for git repos AsciiBinder will sort out how to acquire and unpack the remote file based on the URL pattern per above. Given an entry like this in the topic map, AsciiBinder will behave accordingly: 1. If the remote file/repo is unreachable, AsciiBinder will raise a warning and skip this entry, proceeding through the rest of the topic map. 2. If the specified landing directory _already exists_ in the current branch of the main repo, AsciiBinder will raise a warning and _replace the local contents_. This is intended to enable repo maintainers to explicitly create the landing dir and put a 'DO_NOT_USE.txt' file in it so that contributors know this directory is reserved. 3. AsciiBinder will check to ensure that the contents of the remote file/repo constitute a valid AsciiBinder repo. Assuming we get to this point, the contents of the remote file/repo will be dropped into the current branch of the main repo in the specified landing directory. The topics and topic groups contained in the _remote_ repo's `_topic_map.yml` file will be treated as subtopics / subgroups of the landing directory. ## Use cases Here are some common use cases that this will support. For the sake of these examples, I am not mentioning [distros](http://www.asciibinder.org/latest/guides/maintainer_guide.html#distros-versions-and-sites-_distro_map-yml). Per my proposed notation above, distro flags can be used to affect the inclusion of remote content (or even remote _subcontent_) but these examples are easier to follow without the added complication. ### Include different versions of $remote with related versions of $local I have two repos: * local_repo * v1.1 branch * v1.2 branch * remote_repo * v3.6.8 branch * v3.7 branch And I want to include the remote_repo content in my main_repo docs. Specifically: * local_repo v1.1 should include the remote_repo v3.6.8 content * local_repo v1.2 should include the remote_repo v3.7 content So the `_topic_map.yml` file in the local_repo v1.1 branch will include a record like this: ``` Name: Remote Topic Group Dir: remote_group URL: https://remote_host/repo.git@v3.6.8 ``` And the topic map in the local repo v1.2 branch will include a record like this: ``` Name: Remote Topic Group Dir: remote_group URL: https://remote_host/repo.git@v3.7 ``` But the URLs could just as easily point to archives (tgz/zip/bz) as long as each archive URL corresponded to the right versions of the remote_repo content. ### Include the same version of $remote with every version of $local Slightly different scenario: * local_repo * v1.1 branch * v1.2 branch * remote_repo * master branch (which is also the default branch) And in this case, both branches would have an identical topic map entry: ``` Name: Remote Topic Group Dir: remote_group URL: https://remote_host/repo.git ```
nhr commented 7 years ago

@r0ckarong, @adellape, @indigo423 - any opinions on this design proposal?

adellape commented 7 years ago

@nhr I like this approach, and "link to a zip file that corresponds to a specific version / branch" sounds simple enough.

indigo423 commented 7 years ago

As I'm reading to have the possibility to have git repository as a remote repo sounds like a very powerful enhancement. As I understand the "Caveat" in the case it is a git repository it is using always the default branch which is in most cases "master", right?

With the reference to a specific version to a .tar.gz, I depend on an external process to generate and upload .tar.gz files automatically, right?

The first thought which came to my mind was something like Docker does with a git URL:

docker build https://github.com/docker/rootfs.git#myBranch:myDocFolder

Just as an example we have several git repositories in our project which are independently versioned and contain documentation:

It does not seem to me including remote repos is a way to use AsciiBinder as the main framework for all this type of documentations, cause of the conflict with

Make it easier for content managers to build and publish several variants of the same documentation.

and AsciiBinder forces me right now to be the main git repo for all the documentations including the version branches.

nhr commented 7 years ago

Great feedback, @indigo423

On the git URL notation, you and others have pointed out that there are a number of ways to also specify a desired branch. Above, you are advocating for:

<git_repo_url>#<git_branch>:<filepath_in_branch>

That's as good a notation as any; I will update the proposal accordingly.

To your other point: remember that topic maps are branch specific. So when a topic map includes a remote reference (either a zip/tgz/bz file or a git URL of the type that you specified), we are implicitly saying that this local branch is associated with that remote content. As AsciiBinder processes the local repo, it switches through the branches, and the topic map in each branch may point to a different version of the remote content.

So, in this sense, AsciiBinder can ensure that a given distro/version combination of your local docs is associated with a given distro/version combination of the remote docs if that's what you want. However, you can also set up the remote content URL to always point to an "unversioned" endpoint (like a zip or tar file) so that different versions of your local repo get the same remote content.

I'd like the use of this feature to be applicable to a number of use cases, but I'm also working to ensure that this has no impact on a) basic AsciiBinder behavior and b) users who don't need/want to use this feature.

indigo423 commented 7 years ago

@nhr thanks a lot for the detailed explanation, helps me a lot. I'm really happy to see enhancements like this in the asciibinder project.

Fryguy commented 7 years ago

Regarding the notation, may I recommend the following which more closely resembles the way github denotes references (difference from above is the @ vs #)

<git_repo_url>@<git_branch>:<filepath_in_branch>

See for example how you reference master and how github renders it : https://github.com/ManageIQ/manageiq/commit/master . GitHub doesn't seem to have a shorthand for a file reference, so I think the : is a good delimiter .

indigo423 commented 7 years ago

@Fryguy I don't have a strong opinion about the delimiter @ or #. Just picked the first one I've seen which was the one Docker is using.

In Apache voting speech about @ vs #

+0 :)

bexelbie commented 7 years ago

If I am reading this correctly, I believe this is valid using this proposal:

I have 7 repos, (repo1, repo2, repo3 ... repo7) that contain content. Each repo has the same three branches: f24, f25, f26

I can create repo-master that contains the following branches

I am being so explicit as I haven't had to think about this in a bit, but I know my use case was one of the motivators for this (thank you @nhr).

This seems to get me what I want if the above is valid.

Next we can worry about translations :P

indigo423 commented 7 years ago

Just as food for thought, related to the example from @bexelbie. I can see two strategies from open source projects and how they maintain documentation:

  1. separate docs repository
  2. docs inside of the source repository

IMHO there are pro's and con's and there is no right or wrong to me. If you are something like a "distribution" where you don't own the code like Fedora you need probably something like use case 1. If you are an open source project and you provide docs for your software and do releases you are most likely in something like use case 2. I definitely see the enhancement addressing something in use case 1.

The versioning of the AsciiBinder repository itself becomes for use case 1 and 2 a different meaning. In use case 1 the version of the asciibinder repository reflects how several versions of remote github repositories are orchestrated. In Java developer speech something like a pom.xml where you define which version number of your documentation modules you want in a specific version of the asciibinder repository version.

In use case 2 the version number of the asciibinder repository reflects the documentation for exactly the version of the software it is related to.

nhr commented 7 years ago

@bexelbie yes, that's right. I am trying to cover your requirements plus a similar requirement for the minishift team.

@indigo423 docs inside of a source repo - or more generally, docs that aren't based in the root dir of the repo - will not be addressed in this feature effort. I see that as a notable behavioral change in and of itself, so I don't want to conflate it with other development work. This is to say that I will eventually address that, but not in the scope of this proposal.

indigo423 commented 7 years ago

@nhr yes the scope totally make sense to me. Was just reflecting what I understood so far and where I have to categorize your enhancement proposal. Thanks for the feedback.

bexelbie commented 6 years ago

I think I am putting this note in the right place:

A use case got raised for brining in a repo at a different level of the _topic_map.yml. i.e. this repo build separately for testing, but comes in a level lower than root for production.

vikram-redhat commented 4 years ago

Closing and cleaning up old issues.