Global well known RepositoryMap

[tmspzz]

Enhancement Suggestion

Enable Rome to download & read an well know repository map

1. Keep a well know central RepositoryMap on GitHub
2. Enable Rome to download it on demand (perhaps rome update-well-known [URL] or default URL)
3. When running list, download, upload also check the well known map for framework name resolution

Current and suggested behavior

Currently there is no well-known map.

Why would the enhancement be useful to most users

The enhancement could enable users to write no [RepositoryMap] section of a Romefile

[leonid-s-usov]

Isn't it possible to parse the actual project files in order to extract the target names and avoid the manual mapping altogether?

[tmspzz]

@leonid-s-usov

Fist off, if all your dependencies are of type "github ..." and follow the naming Owner/FrameworkName where FrameworkName is the actual name of the output binary (like Alamofire/Alamofire that produces Alamofire.framework) you don't need any mapping. It just works. Maybe this is not clear from the docs.

The mapping is needed for a couple of reasons.

• To resolve the name of the produced binary when it can't be derived (git repos and binary dependencies can be named whatever, and that is the only trace they leave in the Cartfile)
• To resolve multiple names in case a project produces multiple binaries
• To explicitly ignore some binaries

Now, to your question:

Isn't it possible to parse the actual project files in order to extract the target names and avoid the manual mapping altogether?

The mapping is needed because the project file (and thus the shared scheme) of the dependencies is not available the moment Rome runs.

The flow is:

• You have your app checked out .
• Along with the app the Cartfile and Cartfile.resolved are checked out too
• There is no Carthage/Checkout folder yet because the whole point is not to checkout or build
• You run rome, get the binaries, compile and live happily ever after without waiting for the dependencies themselves to compile.

So it's a bit of a chicken and egg problem.

But ok, let's suppose you actually have the Checkout folder.

Then I suppose it's possible to read the names of the binaries and reverse match from the checkout path to the dependency in the Cartfile.

However I made rome to cover the first, more brutal, scenario.

I have an branch that has work in progress to try to auto generate the Romefile for you from a full carthage checkout if you want to pick it up.

[leonid-s-usov]

Thank you for the detailed response!

Indeed my question was upfront naive, but at the same time I still feel there is a place for improvement. This mapping looks to me quite a bummer to do manually given the overall idea of the tool to be transparent (as I would love to see it) to the whole build process. Looking at my own Cartfile I see at least realm-cocoa which would generate Realm and RealmSwift, and also some indirect dependencies which could not match as well and that would be a tiral-and-error way of coming up with the mapping

Generating the rome file manually would probably solve the problem, although in this case I would put this mapping information in some kind of rome.resolved file to separate this from user configurable settings, because as long as there is a way to get the mapping automatically I don't see any reason for the user to be involved into it at all

I'm thinking that even in the case when there is no Checkout folder and rome tries to get the needed binaries from the cache repository there could be some logic to get the mapping from the file previously pushed to the repository when it was originally created. Apparently rome has already run somewhere else to generate the cache, and if not then this will be the run which will have to build everything. So technically there is no chicken and the egg problem because there is always some rome process which runs initially together with the build (or after the build) and has access to all needed data to generate the clues for itself for later.

Hope this makes sense.

[tmspzz]

@leonid-s-usov Thanks for the interest and for the suggestions.

I'm not trying to defend the current state of things and would be more than happy to make everything transparent if possible.

That said, let's discuss a couple of things.

Generating the rome file manually would probably solve the problem, although in this case I would put this mapping information in some kind of rome.resolved file to separate this from user configurable settings, because as long as there is a way to get the mapping automatically I don't see any reason for the user to be involved into it at all

The configuration file format is INI with sections. The section [RepositoryMap] is not required at all. So a minimal working Romefile looks like:

[Cache]
local = ~/Library/Chaces/Rome

If I understand correctly what you're saying, you're suggesting that the [RepositoryMap] section should go in Romefile.resolved. This is because in your vision the resolved file could be generated by Rome itself. Sure, makes sense if there is auto mapping.

there could be some logic to get the mapping from the file previously pushed to the repository when it was originally created.

You don't have to upload via Rome but let's ignore that for sake of simplicity. When rome uploads at the moment it still uses the map to figure out what to upload and where.

Let's take realm-cocoa example.

During upload the map is used to tell it that BOTH Realm and RealmSwift have to go under the same cache path (i.e. <cache-prefix>/realm-cocoa/<platform>/[Realm |RealmSwift].framework)

If I understand correctly, you're suggesting that since these binaries didn't appear out of thin air, at this point Rome could check the Carthage/Checkout folder, reverse match and generate the Romefile.resolved.

I didn't think about this flow. Seems like it could work and avoid the manual map.

As a candy, I think it would also be possible to upload the Romefile.resolved to the remote cache at top level so that the cache auto-describes itself to someone that doesn't have the Romefile.resolved.

This is a very appealing solution. My estimate of development time for this would be 2-5 days depending on your Haskell foo. At the moment I don't have this time but If you or anyone else wants to step in, be my guests. I have a small refactoring that I would like to push first though.

Now the actual suggestion in this ticket was for a simpler solution, where a [RepositoryMap] would be stored somewhere remotely for all rome users that wish to use it. So that to them, it would appear like "magically" rome is aware of both Realm and RealmSwift.

[leonid-s-usov]

Good, so we are on the same page. Like I stated, I believe this information about mapping should be feasible to generate and consume without the user's intervention. If uploaded to the remote cache I am thinking that maybe the fine should not be even created in the code repository at all, just as some version metadata at the known path in the repository. Could be even combined with the carthage version files into a single data file with the mapping and corresponding versions together.

The reason I posted on this issue is that although it does add this "magic" to the end user, it is laborious and error prone for the cases when the mapping is not up to date or the used carthage packages are proprietary. Someone would need to maintain this repository and it could grow pretty big over time.

Unfortunately, I have seen Haskell up close for the first time when browsing this repository source code, and to tell the truth I haven't felt this click inside to go and learn some Haskell. So I will not promise anything as of now

In any case, watching this project. Thank you

[tmspzz]

@leonid-s-usov Have you seen this issue: https://github.com/blender/Rome/issues/95

This suggests using the .version file information but that would mean that Rome would only work if you use carthage with --cache-builds so I'm not so adamant on the idea. Maybe I'm too cautious.

Can you elaborate on this? What do you mean by "just as some version metadata at the known path in the repository"

If uploaded to the remote cache I am thinking that maybe the file should not be even created in the code repository at all, just as some version metadata at the known path in the repository

[leonid-s-usov]

What you said,

As a candy, I think it would also be possible to upload the Romefile.resolved to the remote cache at top level so that the cache auto-describes itself to someone that doesn't have the Romefile.resolved

and what I said earlier, as well

I'm thinking that even in the case when there is no Checkout folder and rome tries to get the needed binaries from the cache repository there could be some logic to get the mapping from the file previously pushed to the repository when it was originally created

all this is about the fact that the mapping metadata may be uploaded to the repository by the process which initially pushes all data there (or updates), and then consumed by other clients. Not even necessary to have this mapping committed to the source code as the suggested rome.resolved alongside the rome.ini. I just used the naming convention to resemble Carthage layout and point out the fact that the mapping info is purely "technical" and shouldn't be manipulated directly by the user.

Regarding the --cache-builds, looks like I was under wrong impression that the flag was required for rome to work in any case, wasn't it?

[leonid-s-usov]

yep, that issue #95 is exactly to the point! what do you think of maybe making the rome tool call carthage on the user's behalf and make sure that the flags are set correctly? The local cache could actually be used directly be carthage. I did this in my project by creating a symlink of ./Carthage -> /path/to/local/carthage/cache/Carthage

[tmspzz]

The problem with #95 is that .version files are ALSO part of the cache. So can't use them for mapping.

.version files tell carthage not to rebuit downstream depedencies found already built in Carthage/Build, that's all it's does.

Rome is not supposed to call cathage because there is nothing to set correctly :\ It does not depend on --cache-builds

I'm not sure why you're doing /Carthage -> /path/to/local/carthage/cache this if you're using rome. Rome will copy things at the right place for you.

[leonid-s-usov]

I'm doing this because i'm not using rome - yet 😄 . And that's the reason why I'm here, because I was looking for some solution

The problem with #95 is that .version files are ALSO part of the cache. So can't use them for mapping.

Not sure I get it.. why can't the tool use files from the repository prior to making the decision? That is, download the small metadata file (maybe just the version files themselves, if --cache-builds is used) and use it to understand the mappings and then move on with its usual flow

[tmspzz]

@leonid-s-usov You seem to be onto something better than I can think of at the moment so here is the challenge for you to solve:

• If there are no .version files, game over. No mapping possible atm and we all agree on that.
• If there are version files:
  • How does the upload process to cache look like (think of git dependecies not on github and with messed up naming or multiple names)
  • how does the download process look like

Here is the description of what the cache currently looks like for reference: https://github.com/blender/Rome#cache-structure

Ideally the change should not be breaking

[leonid-s-usov]

Thanks for this question, i will try to elaborate later tonight.