search

Metacello / metacello

Metacello is a package management system for Smalltalk
MIT License
87 stars 43 forks source link

Baseline dependency circularity not detected #383

Open dalehenrich opened 8 years ago

dalehenrich commented 8 years ago

Given three baseline specs:

BaselineOfFoo>>baseline: spec

  spec
    for: #common
    do:
      [ spec
        baseline: 'Bar'
          with:
            [ spec
            repository: 'github://peteruhnak/ci-bar:cycle/repository';
            loads: #('plugin') ];
        import: 'Bar'.
      spec
        baseline: 'Zar'
          with:
            [ spec
            repository: 'github://peteruhnak/ci-zar:cycle/repository';
            loads: #('plugin') ];
        import: 'Zar'.
      spec package: 'Foo'.
      spec
        package: 'Foo-Plugins'
        with: [ spec
            requires: #('Foo');
            includes: #('Bar' 'Zar') ].
      spec group: 'default' with: #('Foo').
      spec
        group: 'complete'
        with: #('default' 'Foo-Plugins') ]
BaselineOfBar>>baseline: spec

  spec
    for: #common
    do:
      [ spec
        baseline: 'Foo'
        with:
          [ spec
            repository: 'github://peteruhnak/ci-foo:cycle/repository';
            loads: #('default') ].
      spec package: 'Bar' with: [ spec requires: 'Foo' ].
      spec group: 'plugin' with: #('Bar')  ]
BaselineOfZar>>baseline: spec

  spec
    for: #common
    do:
      [ spec
        baseline: 'Foo'
        with:
          [ spec
            repository: 'github://peteruhnak/ci-foo:cycle/repository';
            loads: #('default') ].
      spec package: 'Zar' with: [ spec requires: 'Foo' ].
      spec group: 'plugin' with: #('Zar')  ]

and the following load expression:

Metacello new 
  baseline: 'Foo'; 
  repository: 'github://peteruhnak/ci-foo:cycle/repository'; 
  load: #('complete').

Metacello goes into an infinite loop:

Fetched -> BaselineOfFoo-PeterUhnak.10 --- github://peteruhnak/ci-foo:cycle/repository [b45e0db:cycle] --- github://peteruhnak/ci-foo:cycle/repository
Loaded -> BaselineOfFoo-PeterUhnak.10 --- github://peteruhnak/ci-foo:cycle/repository [b45e0db:cycle] --- github://peteruhnak/ci-foo:cycle/repository
Loading baseline of BaselineOfFoo...
Fetched -> BaselineOfBar-PeterUhnak.11 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository
Loaded -> BaselineOfBar-PeterUhnak.11 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository
Fetched -> BaselineOfZar-PeterUhnak.9 --- github://peteruhnak/ci-zar:cycle/repository [ca8e8d0:cycle] --- github://peteruhnak/ci-zar:cycle/repository
Loaded -> BaselineOfZar-PeterUhnak.9 --- github://peteruhnak/ci-zar:cycle/repository [ca8e8d0:cycle] --- github://peteruhnak/ci-zar:cycle/repository
Project: Bar baseline
Project: Foo baseline
Fetched -> Foo-PeterUhnak.3 --- github://peteruhnak/ci-foo:cycle/repository [b45e0db:cycle] --- github://peteruhnak/ci-foo:cycle/repository
Fetched -> Bar-PeterUhnak.2 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository
Project: Zar baseline
Project: Foo baseline
Fetched -> (nearest) BaselineOfBar-PeterUhnak.11 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository
Loaded -> BaselineOfBar-PeterUhnak.11 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository
Project: Bar baseline
Project: Foo baseline
Fetched -> (nearest) BaselineOfBar-PeterUhnak.11 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository
Loaded -> BaselineOfBar-PeterUhnak.11 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository
Project: Bar baseline
Project: Foo baseline
Fetched -> (nearest) BaselineOfBar-PeterUhnak.11 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository
Loaded -> BaselineOfBar-PeterUhnak.11 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository
Project: Bar baseline
Project: Foo baseline
Fetched -> (nearest) BaselineOfBar-PeterUhnak.11 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository
Loaded -> BaselineOfBar-PeterUhnak.11 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository
Project: Bar baseline
Project: Foo baseline
Fetched -> (nearest) BaselineOfBar-PeterUhnak.11 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository
Loaded -> BaselineOfBar-PeterUhnak.11 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository
Project: Bar baseline
Project: Foo baseline
Fetched -> (nearest) BaselineOfBar-PeterUhnak.11 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository
Loaded -> BaselineOfBar-PeterUhnak.11 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository
Project: Bar baseline
Project: Foo baseline
Fetched -> (nearest) BaselineOfBar-PeterUhnak.11 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository
Loaded -> BaselineOfBar-PeterUhnak.11 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository
Project: Bar baseline
Project: Foo baseline
Fetched -> (nearest) BaselineOfBar-PeterUhnak.11 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository
Loaded -> BaselineOfBar-PeterUhnak.11 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository
Project: Bar baseline
Project: Foo baseline
Fetched -> (nearest) BaselineOfBar-PeterUhnak.11 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository
Loaded -> BaselineOfBar-PeterUhnak.11 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository
Project: Bar baseline
Project: Foo baseline
Fetched -> (nearest) BaselineOfBar-PeterUhnak.11 --- github://peteruhnak/ci-bar:cycle/repository [6ac72c0:cycle] --- github://peteruhnak/ci-bar:cycle/repository

 . . . (etc) . . .
dalehenrich commented 8 years ago

see this thread for more info

jvalteren commented 4 years ago

Hi @dalehenrich, I hope you're doing well considering the circumstances!

I've found a similar issue creeping up on me when doing Pharo Seaside client work and I hope you might be able to give some hints.

I have a client web application baseline, which first defines a set of external dependencies, like Seaside3, Voyage, NeoJSON, ZincHTTPComponents, etc. Most of these define a 'restrictive' loads directive, to (hopefully) reduce the number of packages being pulled in to the ones necessary.

After that, the baseline defines dependencies on internal projects, like our Fomantic-UI components, our Google API convenience classes, user account management module, etc. Each of these define their own dependencies, (partially) overlapping with the earlier defined external dependencies like Seaside. These are needed to make these internal projects easy to (re-)use.

I have defined the load type #atomic on all baselines.

As you might already guess, loading the baseline results in a lot of duplicate Project-Fetched-Loaded lines logged in the transcript. Fetching BaselineOfGrease alone is logged 99 times! No wonder loading this baseline takes 6 minutes 20 seconds.

Any ideas on the best strategy to improve this situation? I can share baseline and transcript logs if that is useful.

dalehenrich commented 4 years ago

Hi @jvalteren, I am not quite sure what you are looking to "improve"? I

t seems that things are successfully loading for you - which is the primary purpose of Metacello, is that right?

If you are concerned about the number of times BaselineOfGrease shows up in the Transcript or how long the load takes, I'm afraid that that would require a major overhaul (rewrite) of Metacello ... which I am working on:).

If there is something else that needs to be "improved", let me know...

jvalteren commented 4 years ago

Hi @dalehenrich, thanks for your reply. Yes, it is working at the moment and I am grateful for all of the work you have done on Metacello! Back in the Java ecosystem, I was used to Maven with its (admittedly complicated) powerful dependency management. I think I understand how things evolved through Monticello/Metacello configurations to Git baselines (and why) and I see that broad support for different Smalltalk environments is a lot of work. I hope to have some time available over the coming weeks/months to think/work on this myself. Is there a design document/description of Rowan that I can look at?

dalehenrich commented 4 years ago

@jvalteren, Rowan is still evolving ... but is pretty close to where it should be ... Unfortunately I am not a good writer and with the API still evolving I haven't tried to get into details ... However, I can cover the basics for you ...

First off Rowan is based on definitions, but unlike Monticello, Rowan has definitions for packages and projects as well as classes and methods ... Also unlike Monticello, packages are not loadable entities and unlike Metacello dependencies are not defined at the package level. In Rowan, top-level components are the loadable entites. A top-level component is composed of references to other top-level components,nested components, and required project references. nested components have a condition that must match the platform attributes (same as Metacello platform attributes) in order to be loaded.

The components in Rowan basically replace the groups and platform conditions in Metacello.

Unlike Metacello, the Rowan components are objects written to disk in STON. Here is a very simple project with a Core and a (conditional) Tests package. At the moment I have about 56 different example projects and these are documented here.

In Rowan (unlike Metacello), circular dependencies between projects and packages are allowed, because Rowan does not have a load order ... all loads are atomic loads... The Rowan loader creates classes in the right order and methods are compiled after the classes have been created ... so any structure you can create in an image, can be easily expressed in packages, because load order does not have to be determined by the developers themselves.

Projects are loaded by sending a message to a load spec ... Load specs are similar to a Metacell new load expression, but since it is an object not a smalltalk expression, if is easy to manipulate and change what you want loaded by sending messages to the load spec.

Rowan is integrated with git, so when you resolve a project, a git clone is directly created, so the url for the git project is the url you use for doing the clone ... I have built-in support for different repository definitions so I'm imagining that for Pharo there will be an Iceberg repository ...

References to external projects are accomplished by using load specs. spec_0054 is an example of a project with required projects and the projects directory is where the load specs for required projects are stored....

I'm running out of steam (and I may copy this comment to the Rowan readme:), but I should mention that Rowan supports tonel and filetree package formsts ... and the little rowan directory that you've seen should make it easy to attach Rowan to an existing project without disturbing it's usability with Metacello/Monticello/Iceberg/???

Does this give you some idea?

And of course, I would welcome help from other folks in porting this to Pharo or improving/fleshing out the API/fixing bugs etc. ... I would be willing to spend time in skype to help bring you up to speed if you are game:)

I think I've just about hit the point where the overall API is finally at a point where I think everything pretty much makes sense and there seems to be a place for everything (finally:) ... but there is certainly a lot of cleanup and finetuning to be done ...

My thinking has been that it would make sense to get the definition part of Rowan ported to Pharo first ... creation/reading/writing/tests ... this code is the oldest and most stable ... the paint is not dry on components:) ... this would be enough of Rowan to be able to work out the kinks of marrying Rowan and Iceberg ... the GemStone loader code is much more complicated than would be needed for Pharo because Gemstone has multiple users and symbol lists and multiple symbol dictionaries ... but if we go far enough back in Rowan's code base, the project started with a simple loader that could be adapted for use with Pharo ...

If you are still interested, lets plan on talking over skype and see if we can work things out so that you can come up to speed without spinning your wheels ...

jvalteren commented 4 years ago

Thank you, @dalehenrich! This is a very substantial introduction into Rowan :-)

I am certainly interested to look into Rowan, for use in our enterprise application development workflow. I'll drop you a line by email so we can meet online.