search

jart / cosmopolitan

build-once run-anywhere c library
ISC License
17.81k stars 609 forks source link

Compiling Lua #61

Closed ahgamut closed 3 years ago

ahgamut commented 3 years ago

I thought it might be fun to compile a portable version of a language which has C source code.
I picked Lua, and I eventually got it to compile and run on Windows/Linux.

I thought some of the compilation errors might be useful info:

jart commented 1 year ago

I agree but there's no incentive to creating app store for binaries which would be costly to closely monitor for abuse.

paulwratt commented 1 year ago

I agree but there's no incentive to creating app store for binaries which would be costly to closely monitor for abuse.

@niutech unfortunately the nature of APE binaries means they are easy to abuse. Even with vetting and hashes, to detect something that is intentionally hidden is not a simple task to automate. Just look at how complex JavaScript hacks can get, and those are plain ASCII text files.

From what I have seen in the repo, on the website, and on the Issues, it is not hard (or large) to create your own binaries, outside of those official download locations. To put this in a more familiar context, if we were talking about sandwiches, and I was (your) a parent, you know what my answer would be, right?

It may happen that the possibility of a store type repo with automated protection may become a reality, but that will be far of in the future and requires some foundation work which does not (and will not) exist ATM (I have a TODO that may assist in getting that off the ground, but its not likely to show results in the next 12 months - "my plate has just been filled up" ATM)

jart commented 1 year ago

APE is no different from any binary executable. It's harder to abuse the APE format because it stands out and bad guys want to blend in. Many of the features bad guys want, Cosmopolitan Libc doesn't have.

Right now our App Store is https://github.com/shmup/awesome-cosmopolitan which is a curated list of Cosmo/APE projects we created collaboratively on our Discord. As for burggraf, I don't know who he is.

niutech commented 1 year ago

@jart APE is no different from any binaries, yet app stores for them exist, e.g. AppImageHub, although I agree that it requires additional work in preventing malware. But maybe there is a free C++ compiler in the cloud, where newbies could easily clone an APE repo with Cosmopolitan libc and build it by themselves, ideally by one click, without installing GCC, GNU make and all the toolchain on their PC - something like StackBlitz for Node.js? Or a free CI, which would produce APE binaries from multiple source code repos? That way we would make sure that APE binaries are made from the original open source code. FYI, @burggraf is Mark Burggraf, Senior Developer at Supabase.

paulwratt commented 1 year ago

where newbies could easily clone an APE repo with Cosmopolitan libc and build it by themselves, ideally by one click, without installing GCC, GNU make and all the toolchain on their PC

You just described a Docker image.

@niutech what you are describing is a custom build. The point @jart was making about her binaries is that they are a know quantity. You need to remember that in all honesty, if the single package bin (executable) is working for you, you dont need to update it. This mean there is practically no need for an update process.

Part of the original idea of a store, was to move the package visibility out of the confines of a repo. Unfortunately, the reality of the situation is that no store has got it right. To the point that the previous version of Ubuntu did not support Snap (out-of-the-box) and they designed that store system themselves. Ftalpak is not installed by default either.

I'll quote Explaining Computer regarding Snap!:

Whilst using Snap! rather than other technologies is potentially beneficial and transparent to many users, other find it controversial.

Take a step back for a minute and ask yourself why and how you ended up at APE & Cosmopolitan.

I understand the energy generated by the realisation of what APE, Cosmopolitan and Redbean can do, and spreading the word or giving it greater exposure is one of the 1st results of that. But I see it in the same light as SerenityOS, its not for mainstream (ie fast food) consumption. People who need to use it will end up finding it. People who contstantly need to update there software because someone else develops it, already have a huge selection of Apps and Locations to get them.

If you keep finding new and amazing APE developments or repo locations, and you want to "spread the word", I'd suggest submitting PR's to the awesome-cosmopolitan repo, not just add suggestions to the Issues there. You dont need top install any tools to do that, you can do it all online. Maybe over time it will expand into something that more resembles a store.

On the other hand, I for one (without an x86_64 machine) would appreciate any effort to provide an (online) on demand APE App compiler, if you would like to give that a crack. Particularly if you are interested in a union between MicroWindows and APE binaries, it might contribute to the vision you have for APE distribution and availability.

jart commented 1 year ago

Whilst using Snap! rather than other technologies is potentially beneficial and transparent to many users, other find it controversial.

Take a step back for a minute and ask yourself why and how you ended up at APE & Cosmopolitan.

@paulwratt I adore you for saying this.

On the other hand, I for one (without an x86_64 machine) would appreciate any effort to provide an (online) on demand APE App compiler

We're considering doing it because it's really hard to get a GCC toolchain that works for us on non-Linux. I've never felt fully comfortable recommending things like Homebrew packages which are maintained by people I don't know. Compiling GCC is frightfully difficult, and when it does compile, there's often minor incompatibilities or new linter warnings introduced between versions that cause folks to have a negative experience. What we ideally want is Cosmopolitan Libc to be good enough that it's ready to build a legacy codebase as complex as GCC and let it work the same on Windows and MacOS. But we're not there yet.

What I'd like to have is a 200kB APE binary that just asks one of the Linux servers I operate to build it for you, using our standard toolchain. That way if you use something like MacOS or Windows you won't need to setup a Linux VM inside WSL/Docker/VirtualBox/etc. before you can hack on Cosmopolitan.

The value prop is (1) is you'd only need to curl cosmocc.com, and (2) we can make builds go faster since we're pooling our resources, can cache things, and can have more cpu cores put to use, since right now our workstation cores are mostly idle while we're coding.

AppImageHub

@niutech That's an anonymous unlisted business with a bunch of white-labeled websites making false claims. I'd be surprised if anyone trusts them as a distribution platform for legitimate open source binaries.

But maybe there is a free C++ compiler in the cloud, where newbies could easily clone an APE repo with Cosmopolitan libc and build it by themselves, ideally by one click, without installing GCC,

That's already the case. If you git clone the cosmopolitan repository, it builds hermetically using a vendored toolchain. The git commits here are signed by my GPG key. It doesn't have dependencies. You don't need to build your artifacts in my cloud to have assurances about provenance. Our project is easier to build than any other project you'll find of similar complexity. While I'd love to offer more online services, we'd need to identify an opportunity that makes economic sense.

burggraf commented 1 year ago

Burggraf here :) I created https://github.com/burggraf/awesome-cosmo about a year ago just as a fan page. If you're maintaining a more up-to-date repo for that now, great. I'm a huge fan of your work, @jart. Thanks for what you're doing.

jart commented 1 year ago

@burggraf Thanks for stopping by and thanks for your work promoting the project last year! Since then we've started a Discord community and we'd be honored to count you as a member. https://discord.gg/Z4PXmtwD

paulwratt commented 1 year ago

( hmm.. my reply go lost somewhere. I can see it in my inbox )

On the other hand, .. We're considering doing it ..

my comment was more of an "anti-spirit crushing" out ;)

(but is also a side-interest of mine - I'm doing something similar for m68k-atari-mint)

it's really hard to get a GCC toolchain that works for us on non-Linux.

maybe someone could have a look at the rust-musl-cross docker image spinner, the 1st step of which is compiling a gcc-musl (here) - There might even already be a a Windows binary builder image somewhere, as he is referencing Azure in the final docker update step at the end of the log - also I was just reading another cloned gcc (v9.3.1) repo that had specific MinGW patches for Windows if anyone is interested (btw I am pro- vcc, just so you know)

What I'd like to have is a 200kB APE binary that just asks one of the Linux servers I operate to build it for you

that sounds a lot simpler from a user point of view. BTW I check that Workers server you use, but is seems APE is not possible there (yet), right?

I think over-all (and all round) maybe some more thought about how and what a user that doesn't (or can't) have a toolchain might want out of APE, Cosmopolitan, or Redbean.

But I also think its too soon for something substancial, outside of the awesome-cosmo repo - maybe it can incorporate more specific links (which is what @burggraf had right?), or sub sections - you know, evolve - which is also why I made that suggestion - and while thats happening others can mull over what missing to be able to provide a "give me this app now" APE (and that could become a psuedo- store)

Actually maybe thats the greater part about what was off putting in @niutech language, the context of "what constitutes a store" and "how a store is presented", so maybe the question should be "how to present a store to APE users, that does not look or feel like a store, and that is secure (unlike you average store)" - in which case @niutech might have managed to get some answers

mingodad commented 1 year ago

Out of curiosity what prevent the gcc compiler inside this repository to be built like redbean ? I mean a standalone executable where all the header, object files needed to compile link are inside a zip file attached to it.

paulwratt commented 1 year ago

yeah, see now thats lateral thinking for you :) if it could at least do something like:

cosmocc.com --tiny myapp.c

and reference that appended zip internally without needing to unpack it (ie a re-binadable/extensible virtual fs - PhysicsFS by Icculus) that could help simplify an initiation into the world of APE's, and provide more advanced users and projects better fs layout opportunities

mingodad commented 1 year ago

I did an experiment trying to compile/buil c2m from https://github.com/vnmakarov/mir and got it to build but when trying to use it then several issues arised see here https://github.com/vnmakarov/mir/issues/297

paulwratt commented 1 year ago

On the other hand, .. We're considering doing it ..

@jart your busy atm .. let someone else have a play, come back to it when things slow down a bit (unless you can keep an eye on where the "someone else" is going with it) - my comment was more of an "anti-spirit crushing" out ;) (but is also an interest of mine)

it's really hard to get a GCC toolchain that works for us on non-Linux.

maybe someone could have a look at the rust-musl-cross docker image spinner, the 1st step of which is compiling a gcc-musl (here)

What I'd like to have is a 200kB APE binary that just asks one of the Linux servers I operate to build it for you

that sounds a lot simpler from a user point of view. BTW I check that Workers server you use, but is seems APE is not possible there (yet), right?

I think over-all (and all round) maybe some more thought about how and what a user that doesn't (or can't) have a toolchain might want out of APE, Cosmopolitan, or Redbean.

But I also think its too soon for something substancial, outside of the awesome-cosmo repo - maybe it can incorporate more specific links (which is what @burggraf had right?), or sub sections - you know, evolve - which is also why I made that suggestion - and while thats happening others can mull over what missing to be able to provide a "give me this app now" APE (and that could become a _psuedo-_store)

Actually maybe thats the greater part about what was off putting in @niutech language, the context of "what constitutes a store" and "how a store is presented", so maybe the question should be "how to present a store to APE users, that does not look or feel like a store, and that is secure (unlike you average store)" - in which case @niutech might have managed to get some answers

A build on demand would be great, but APE's are static, you dont need to have versions (or maybe you do, based on Cosmopolitan repo updates) - except maybe in certain circumstances, like say RedBean - but even then its not like a must have based on what that is being used for already

( I replied in an email, I'm going to submit and review this a bit more, got longer than I expected )

jart commented 1 year ago

Anyone is free to set up their own platform for distributing APE binaries written by people other than themselves. I have no authority. However it'd be unlikely to earn my support, since if such a thing were to happen, it should be done by the person who created APE. In order for me to be interested in doing it, I'd want to see more people who are using APE to ship their own programs that they've written, on their own. Then once enough people have adopted APE as their preferred format for shipping their releases, then we can build a platform since it'd be solving a logistical problem for our users. Until then I'm happy to focus on creating better build tools, since they'd make APE more attracting to folks adopting it for releases.

jart commented 1 year ago

With that said, app stores aside, here's something that anyone can and should do. You can build your open source distro with APE. Our open source conformance just hit an inflection point yesterday that allows it; see https://github.com/jart/cosmopolitan/commit/e557058ac859691fb21a91d5e9281e9fc9e6b165. If you read our toolchain instructions, you'll notice:

https://github.com/jart/cosmopolitan/blob/31dab8a75d928df7a4cd2e9bcf42916b1810d085/tool/scripts/cosmocc#L17

This /opt/cosmos directory is where software goes after you successfully get it to build and run make install. I think we should turn this into a game where we all try to port as much software to cosmopolitan for ourselves, and then each of us can upload our /opt/cosmos folders to our own web servers with directory listings enabled, or in tarball format, binaries and all, as proof that we did it. Then anyone in the community can download your binaries and run them using the pledge.com sandbox tool.

I think this would be a fun game. If the game is fun enough, then it'd potentially grow up and become its own Linux-distro, except rather than Linux it'd be a meta distro, that each program is its own operating system that runs on all operating systems.

jart commented 1 year ago

that sounds a lot simpler from a user point of view. BTW I check that Workers server you use, but is seems APE is not possible there (yet), right?

It's possible to upload APE binaries to Cloudflare workers. If you're referring to https://worker.jart.workers.dev/ then I that's my CDN for image files and .com.dbg files. I put the .com.dbg files on there because the DWARF data is huge, they're mostly only downloaded by crawlers, and Cloudflare R2 offers free network bandwidth egress. I use GCE for the official domains and Google sends me outrageous bills when stories make the front page of Hacker News, which isn't sustainable for a free project like this. However I keep the .com files on the official domains because I want people to know for certain the software is coming from the source they trust, just in case anyone doesn't know I run the workers domain.

mingodad commented 1 year ago

About gcc with musl I found this https://musl.cc/ that have several read to use distributions of gcc and uses https://github.com/kvinwang/musl-gcc as base to build then.

stefnotch commented 1 year ago

@jart I would have loved to check out the Discord server, but the invite link seems to have expired. Is it possible to get a new one?

(Also, on a somewhat related note: There's this lovely blog post from someone else, which also seems to have an expired Discord link https://ahgamut.github.io/2021/07/13/ape-python/ )

jart commented 1 year ago

Here's the latest Discord invite, which expires in seven days. https://discord.gg/vFdkMdQN