The iPKG format should be replaced with a known configuration file format.

jfdm commented 9 years ago

For now the current ipkg format has served us well, however, with the interest in developing an Idris package manager the format should be replaced with a known configuration file format i.e. YAML.

YAML appears to be the choice de jeur for configuration files. Use of YAML will also allow for definition of a checkable schema for package files.

There are several advantages for switching to a known format. For one, it appears to provide the benefits of XML sans the horror's associated: extensible, well formed, types and namespaces, CDATA et cetera It is essentially, key-value pairs but on steroids and is more human readable than JSON. Secondly, support for a bespoke configuration file parser can be deprecated and work concentrated on using the contained values. Discussions over syntax also becomes deprecated to a degree.

The only disadvantage would relate to the Haskell library used. For example: not all libraries support schema checking, some require external dependancies and unsure if said dependancies are cross-platform. The PKG file parser and format need to be rewritten.

An example iPKG YAML file. Note the default options are explicitly represented here, and need not be represented.

This issue should be seen as a place holder for work in progress by myself. Though, if someone else completes this before me I won't be mad.

suhr commented 9 years ago

YAML seems to be pretty complicated.

What about TOML? Rust's Cargo uses it, for example.

jfdm commented 9 years ago

I haven't heard of TOML. Can you clarify more, why you think TOML should be considered over YAML.

suhr commented 9 years ago

TOML config example: https://gist.github.com/suhr/2a4747c3b5b3c1062094

I haven't heard of TOML.

https://github.com/toml-lang/toml

Can you clarify more, why you think TOML should be considered over YAML.

The main reason is it's more simple.

ahmadsalim commented 8 years ago

@jfdm Do you think it is still relevant to change format?

jfdm commented 8 years ago

At the moment I do not think we should change the format it is pretty stable after all, but I am still worried about extensibility and maintaining a custom parser. At some point in the future we should revisit it certainly.

LeifW commented 8 years ago

I have work adding support for a yaml format in addition to the current ipkg format I could push up. I had been hoping to have other tools read the files e.g. dependency managers. On Apr 11, 2016 11:51 PM, "Jan de Muijnck-Hughes" notifications@github.com wrote:

At the moment I do not think we should change the format it is pretty stable after all, but I am still worried about extensibility and maintaining a custom parser. At some point in the future we should revisit it certainly.

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/idris-lang/Idris-dev/issues/1825#issuecomment-208735585

langston-barrett commented 7 years ago

@LeifW Any progress? It would be great to move to a standard format, rather than maintaining a custom one.

jfdm commented 7 years ago

I don't know about the work of @LeifW, but I do have something in the pipeline. Sadly, work commitments take priority, and won't get back to it until beginning of November.

LeifW commented 7 years ago

I can put up a PR for what I have later this week or this weekend. Jan looks like he's put more thought into this than I have, but perhaps my work can be helpful as a starting point.

be5invis commented 7 years ago

+1 for TOML. It is simple enough

pierrebeaucamp commented 7 years ago

Any thoughts on https://github.com/sol/hpack?

wizzardx commented 6 years ago

Hi there.

How about taking a page from how Nim and Nimble handle package management?

The'y're not perfect, but perfect can also be the enemy of good.

Central list of packages is maintained in a json file on github:

https://github.com/nim-lang/packages/blob/master/packages.json

People who want to add packages to the central listing can do a git pull request against the JSON file.

Each package is usually just a git or mecurial repo, but needs to contain a .nimble file with more details, eg for their SDL2 wrapper lib:

https://github.com/nim-lang/sdl2/blob/master/sdl2.nimble

The packages themselves are also listed nicely in the docs, pulled in via javascript to stay current, eg:

https://nim-lang.org/docs/lib.html#nimble-unofficial-packages

https://github.com/nim-lang/nimble

And then the package manager; it downloads the json file and uses that metadata.

It's fairly basic, but works pretty well in practice.

How about making a really basic idris package manger; eg "idris-pacman", that for v1 just uses that kind of basic scheme?

xaviervia commented 6 years ago

Configuration files are often manipulated by tools written in a very diverse set of languages, which means there is a clear plus in making them as interoperable as possible. Having a simple, widely known format for the configuration file makes it easier for an ecosystem to flourish.

With that in mind, I propose to go with JSON.

Every language in existence has a way of handling JSON, and it is not going away any time soon. It might not be the prettiest, but that is offset by the fact that there are so many tools to consume it, produce it and manipulate it.

wizzardx commented 6 years ago

There's some precedence there: The D lang packaging format:

https://code.dlang.org/package-format?lang=json

eg:

{
    "name": "myproject",
    "description": "A little web service of mine.",
    "authors": ["Peter Parker"],
    "homepage": "http://myproject.example.com",
    "license": "GPL-2.0",
    "dependencies": {
        "vibe-d": "~>0.7.23"
    }
}

xaviervia commented 6 years ago

That's verbatim the same as Node’s package.json format. And there is something to be gained from sharing practices: after all most of the keys in the package.json format from Node are general purpose description of programming libraries and have nothing specific about JavaScript or NPM in them. @wizzardx , I’ve seen that same format being adopted more and more for project manifest files outside Node.

I know I’m biased coming from a web-heavy background, but I time and again I find that nobody has a problem with having to deal with JSON when coming to the web since everybody that has been programming for a while bumped into it before and knows how to use it.

jdevuyst commented 6 years ago

How about making Idris the file format?

When I propose this, I’m inspired by Boot, which is build tool for Clojure where the build file is also written in Clojure.

Here’s how this could work. The package.ipkg would be a regular Idris file except that it has an entry point package : PackageInfo.

PackageInfo would be a record type that contains the program name and other metadata. Additionally it could have various IO values, which are ‘tasks’ that serve various purposes:

Download (and potentially patch?) dependencies
Compile with various configurations
Run tests
Start the REPL (perhaps with some code injected on launch)
Run the Makefile
...

It might then of course help to add a bunch of ‘batteries’ to make writing these tasks easier. The batteries could be convenience functions for downloading and installing packages, for launching a REPL, etcetera.

I do imagine that safety could be a concern. However:

Safety at compile time is already non-existent given that ipkgs may cause make to be called and given that type providers allow I/O to be performed during compilation.
If we want more safety pre-compilation then perhaps that could be solved by adding a ‘safe’ mode to Idris. In this mode really_believe_me, assert_total, unsafePerformIO, and so on might be disabled. The ipkg file could be evaluated in such a mode, meaning that the only unsafe operations would happen through the IO monad (and if that’s still too dangerous, we could add a SafeIO monad).

mkoloberdin commented 6 years ago

Might be useful:

luc-tielen commented 6 years ago

I've been tinkering with a "package manager/build tool" of sorts for Idris, but I would really love a standardized format (or natively written in Idris) that's properly supported by the language itself.

This is one of the main things preventing me from building larger projects.. I like @jdevuyst his suggestion, that or the YAML/TOML approach..

david-christiansen commented 6 years ago

The more I learn about YAML, the less it seems like a good idea for something like this. YAML is extremely complicated.

The approach of internalizing the metadata proposed by @jdevuyst sounds ambitious, yet promising. Otherwise, TOML is quite a bit simpler than YAML, while still being friendly for humans. Something simpler than that would be even nicer, as we probably do want to be able to write verified Idris code to process these things in the future, but OTOH the best parser is the one you don't write yourself.

LeifW commented 6 years ago

You don't have to use all of YAML, just a simple subset. YAML is already used as the package format by a lot of related tools, e.g. stack, hpack, etc. I have a branch that uses yaml instead of .ipkg, and I like how it looks. I guess TOML is used by rust, but I don't have as much experience with it. Ultimately, I'd like something extensible, so I can add other metadata to project files. And also easily parseable by other tools.

david-christiansen commented 6 years ago

The problem with picking a simple subset of YAML is that you're no longer using YAML, so any benefits associated with a reusable format go out the window. It becomes yet another proprietary format, with extra gotchas associated with the difference between how it seems and what it actually is. When written tastefully, YAML definitely looks good, but it's really a very complicated format.

Extensibility is a good goal, however, and whatever we get should support that. If we do get an Idris value representing a package, it'd be pretty easy to render it into whatever format was convenient for any old external tool, wouldn't it?

jfdm commented 6 years ago

As the original submitter of this issue, I have been musing it for quite a while. My own, publically available, build/packaging helpers for idris do use yaml. I even have a half finished branch which replaces Idris' own format for YAML. I think i was stuck on parsing namespaces.

I too have some misgivings about using YAML, exactly what @david-christiansen has mentioned, but I think we can work with the format regardless.

However, I am not a fan of encoding idris' package metadata in the language itself.

I'll see if I can polish off the half finished branch over the next few weeks and see where I am at. I don't want this issue just being about discussing things, it would be nice to see some action.

david-christiansen commented 6 years ago

I certainly don't want to get in the way of work here that people will find useful. I've registered my concerns.

doofin commented 6 years ago

I strongly agree with @jdevuyst . Scala 's sbt and new build tool mill is also build around this idea. No need to use other file format, just idris.

doofin commented 6 years ago

If you want write idris everywhere,check https://github.com/idris-industry/ikan

f-f commented 6 years ago

I was wondering about Idris package management and ended up on this issue, so I'll add my perspective (sorry for not contributing more practically): some projects are adopting Dhall (Haskell implementation/bindings here) as the language for their package management, e.g.:

Eta (Haskell on JVM) will have it as the default language for build config, replacing .cabal, see here
PureScript: there has been discussions about supporting it in psc-package (the main package manager, currently only JSON) and there is a new package manager entirely based on Dhall, Purcel

You can think of the language as "JSON + functions + types + imports", and it's not Turing-complete - so safety is not a concern anymore - and can easily render to JSON/YAML/text/etc.

(Disclosure/disclaimer: I'm a contributor to the lang)

I used the boot-clj tool described above to build some Clojure projects, and in the end I stopped using it, as there is some drawbacks to the "single language for everything": basically you're now writing in a full-fledged programming language and the build/package config is an actual program, so its logic requires maintenance, can get too complex and need refactoring, there are security concerns, etc etc.

ahmadsalim commented 6 years ago

I have seen a lot of discussion about approaches in this issue. While we appreciate a good discussion, it would be nice to move forward with some approach.

If someone is genuinly interested in making a PR to a particular language, please start a discussion on the mailing list or on IRC. It's probably easier to guide people using a particular approach, and then we can get closer to closing this issue 😄 .

It might be interesting to have something that works well with Blodwen, which is most likely going to be the next version of Idris.

clayrat commented 6 years ago

idris-lang / Idris-dev

The iPKG format should be replaced with a known configuration file format. #1825