travis-ci build

[LADSoft]

now that github can trigger travis-ci builds again, write a build-script for the travis-ci build. this is ONLY to build once with an alternate compiler and is not intended to compile the compiler with itself. Challenge is it has to run on linux and should probably use GCC. It is assumed something similar to the current scripts could be used as omake is somewhat compatible with gnu make.

[GitMensch]

You may want to not only define but also add two internal variables to the Makefile and sub-Makefiles: $(COPY) (copy or cp) and $(PATHSEP) (\ or /) and additional in sub-Makefiles: $(MKDIR) (mkdir or mkdir -p)

$(PATHSEP) should fix the current issue.

[GitMensch]

$(MKDIR) should fix the now current issue - the normal unix mkdir doesn't create the missing child directories, but mkdir -p will; for portability $(PATHSEP) would still be reasonable. And maybe investigate why $(NULDEV) evaluates to NUL instead of /dev/null

[LADSoft]

wish i had read your comments before working on it... could have used mkdir -p. that $(PATHSEP) idea is looking good about now too lol! Maybe I will rework that so it is cleaner, after I get further along.

right now my main problem is with goofy things like random inclusions of windows headers and #include file names not having the right case.

[chuggafan]

Yhea, that's a lot of why I wanted things such as c++11 full threading support: it makes supporting linux SO much easier because you're removing windows header dependencies

[GitMensch]

... actually I was a bit surprised that you do all this in master branch. That is no issue, I just was expecting you'd do this in a ci-branch, do the complete cleanup and then a single commit to master. But after all those commits it lost its minor importance completely...

wish i had read your comments before working on it... could have used mkdir -p. that $(PATHSEP) idea is looking good about now too lol! Maybe I will rework that so it is cleaner, after I get further along.

That was the single reason I've posted them... And yes: doing a cleanup in the Makefiles is surely useful as this would lower the "special" cases to a minimum and make them easier to maintain.

right now my main problem is with goofy things like random inclusions of windows headers and #include file names not having the right case.

In any case it is good that you fix windows-dependencies where they exist by using the correct casing in includes and remove them where they aren't necessary like the change to system().

This definitely is worth the work, even in the parts that aren't ci-related!

[LADSoft]

the change to system() is necessary on windows, because some of the command lines omake generates while building the project are so long the built-in system() can't handle them (because of line length limitations on cmd.exe). Don't know if linux has the same problem... but really my goal for travis-ci is just to get it to compile so I can proceed with coverity. At least for now!

[chuggafan]

A good way to fix the issue with the omake command lines is to allow it to link it in parts, e.g. 1 part gets linked together to form a bigger blob that then gets linked together, done conditionally if it detects the whole thing would be longer than the command line length (8091 characters on anything XP and after), this would allow for a simple system to build it with the command line path limitations, however, this would be an interesting challenge to fix anyways.

[GitMensch]

A good way to fix the issue with the omake command lines is to allow it to link it in parts, e.g. 1 part gets linked together to form a bigger blob that then gets linked together, done conditionally if it detects the whole thing would be longer than the command line length

Do you think about this part as something omake could fix? If it is only about linking and omake would use an inbuilt rule for this, this may could work. Otherwise it looks like those long command lines should be splitted (also because of needed time to rebuild if only a part was changed) into groups within the offending Makefiles.

[LADSoft]

I was trying to think and it seems there is another reason for the SYSTEM() rewrite... I still support DOS (I keep trying to get rid of it but then get a lot of FLAK) and unfortunately FREEDOS was broken last time I looked and there was no prospects of them fixing it. Basically, I had to go through the HXDOS spawning rather than try to use command.com, because any attempt to use command.com with a standard version of system() just didn't work, with the borland compiler. Other DOS variants worked properly though... I pinned it down and told them what to fix but they didn't seem interested...

but as far as the fact of the 'huge' links omake is generating those in the first place. The original plan was to use a file-oriented thing sorta like what borland make does (you do @&&| and then end it with | and it puts everything in between in a temporary file and then puts the temporary file name on the command line). The tools all support huge files for that. But gnu make doesn't support it so I had to quit with that approach last spring when we were doing all the omake updates. The main solution would have to be, rewrite the make files to glob things together in smaller chunks and then do partial links (the linker will support it). But that will add specificity to the make file that i didn't really want to add... and there is no ability to make syntax changes to support a more general approach because of the gnu make compatibility problem. And the only other thing is to have omake detect the specific linkers that are likely to be involved and split lines accordingly (if it is even possible with all the linkers involved).

If anyone has any other ideas I'm open to them...

[GitMensch]

Congratulation for getting GCC and Travis working! I suggest to handle the Makefile cleanup in this ticket and then close it as it works now "build passing :-)".

Coverity has its own issue #63 so "travis with coverity" may also be handled there.

[LADSoft]

i did most of the cleanup the other night... just a min as there is one left...

note that I did get pretty much everything compiling, with the notable exception that I had to exclude OCIDE... too much direct dependency on windows.

[LADSoft]

unfortunately the changes for the travis build broke the appveyor build, hope to have it working again tomorrow...

[LADSoft]

at this point the continuous integration seems to work on both servers. I'm closing this.

[GitMensch]

We may open a new ticket for this but I'd suggest to reopen this one instead...

Since October 2018 Travis CI added Win32 environment, along with Visual Studio 2017 build tools, so it should be possible to use this one to also compile and run the existing regression tests on Win32 by only adjusting the travis configuration file.

[LADSoft]

I tried to do win32 as that was preferred to having to get it working on linux, but couldn't get it to work. I'll try again if you think they added it just last month...

[GitMensch]

I'll try again if you think they added it just last month...

Following the notes in the given Travis-CI blog entry should work. Please use a multi-os variant, keeping the parts that are currently working under linux host. Extending the commands done in win32 (up to what we do on AppVeyor) will need a check for the OS, but should be doable.

[LADSoft]

ok as it stands I've got what I think might work for the travis CI on a combined windows/linux, however, there are bugs in travis ci that don't allow me to complete it. Specifically, chocolatey won't work properly in the presence of secure environment variables. This seems to be emblemantic of a deeper problem with secrets in the windows environment in general.... I'll keep an eye on it and continue with this once they've got it resolved.

[LADSoft]

as far as the appveyor builds, I finally got all my ducks in order but then ran out of disk space on the appveyor server... trying to resolve...

[GitMensch]

As the release is out of the door I wanted to ping on this again.

[LADSoft]

I actually checked this a couple of weeks back, I couldn't find any updates that seemed to mean they might be working on it and left it at that.

So today I've tried the windows build again, it fails in the same place. The build tag still says windows support is in the very early stages as well...

guess we have to continue to table it for now.

[GitMensch]

guess we have to continue to table it for now.

OK, I'll drop a note here if I read something "new". Converning CI we also still have #255 and as it is said to be much faster than Appveyor it is likely useful to implement it, too.

[GitMensch]

Update: Windows on Travis should work relative well now. See documentation and open issues (none of these seem to be important for this repo). If anything is not clear afterwards then there's the invitation to ask at the linked discussion board.

[LADSoft]

i think i like travis a little better. than appveyor... if I can ever get the buidls to work lol!

[LADSoft]

well this was a nice idea, but it turns out the build time-out on travis-ci.org is 50 minutes. The individual builds definitely take longer than 50 minutes on that platform, so, we cannot do it. I will remove the logic to invoke the builds on windows...

[LADSoft]

note that the posix-seh rev of mingw64-mak built executables that had problems compiling the library, with omake -j:4. We are elsewhere using the w64 rev.

clang also had problems; I use clang rev 9 with success but excutables built on clang rev 10 may hang when building the library. I will probably open a separate issue to investigate further.

[GitMensch]

note that the posix-seh rev of mingw64-mak built executables that had problems compiling the library, with omake -j:4. We are elsewhere using the w64 rev.

So I guess there's an issue for this?

The individual builds definitely take longer than 50 minutes on that platform, so, we cannot do it. I will remove the logic to invoke the builds on windows...

Hm, so Travis now builds on linux with GCC as one build, correct? Can you please revert 8f374316962a0b5fc8d54dc7f6b9c4686a778af1 and comment the lines out instead of removing them? That would be useful to be able to re-investigate if things change in the future and also would provide a template for other environments like the git workflow.

And then - Why does the build take so long at all?

[LADSoft]

yeah travis now builds with GCC.

I readded the llines in .travis.yml but commented them out this time. Will push it later...

As far as why the builds take so long I dunno. I haven't done measuring but it seems like they complete more quickly on my home computer. I also have been making various fixes this year that seem to have the qualitative effect of speeding up builds on my home computer, but the builds on appveyor never change in speed no matter what I do at home. Except for minor fluctuations based I assume on appveyor load. I don't know why; assume it has something to do with the virtual machines but just don't know.

[LADSoft]

just for some closure I timed a build at home... took 20-25 minutes with omake -j:8. I did some experimenting and appveyor doesn't improve after -j:2, but -j:2 is an improvement over -j:1. Appveryor reports as having two processors, so...

It crosses my mind that the latest builds on travis were with -j:2, maybe I will round this out with an experiment with a higher number to see if the timing improves 😄

[LADSoft]

I tried Travis with -j:8. It didn't change anything. Travis gets into running the tests and then aborts on a timeout... I also got a report on the number of processors; Travis also reports two processors. At this point the obvious way to get Travis working is to cut out the third compile (the time is currently for three separate compiles of the suite plus the test run) but I don't know if it is really worth running the tests that way.

[GitMensch]

Then maybe remove the test part for those long-running environments? This would at least run one test - the compilation of itself.

[LADSoft]

if i remove the tests it is right on the edge... maybe the clang wouldn't work and maybe it would be a rocky road to work reliably. but I will try again after I complete #436 as that might make some difference as well, who knows?

[LADSoft]

so I tried it again without the tests. One of the builds succeeded with 20 seconds left on the clock; most of the others failed. I've pulled the windows support again...

[GitMensch]

Sounds reasonable, at least we know where we stand on this now and do use Travis for one environment. GitHub Actions next?

[GitMensch]

Maybe reopen that, because travis-ci.org does no builds any more. If I understood it correctly to bring travis for this repo back @LADSoft possibly only needs to activate that at https://travis-ci.com/github/LADSoft/OrangeC and adjust the badge reference in README to point to it afterwards.

[chuggafan]

The problem is travis-ci actually changed their handling of open-source entirely so I'm not sure this would have any advantage over, say, azure pipelines for a linux target...

[LADSoft]

i fixed it. Supposedly, builds are a lot faster on that platform now. Guess I will reopen this though and try win32 compiles again...

[LADSoft]

looks like travis ci will happily let you migrate your account, but the service isn't free any more.

[GitMensch]

Looks like those extra steps need to be done: https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com/#existing-open-source-repositories-on-travis-ciorg

[LADSoft]

yeah after a lot of head scratching I got signed up for the beta, got the app installed on github, and got migrated. But when it came time to build travis put a note in the builds window that I wasn't allowed to build. It did it both for pulls and for builds I tried to trigger from the UI. So I looked and found an article somewhere where someone was saying they had been through all that too, only to find that travis had posted that the service isn't free for open source any more. Originally they gave you a few free hours to kind of entice you to buy but even that has stopped now.

[GitMensch]

Hm, I recently read somewhere about "if your few hours are not enough for your OSS project contact us with a note about how much CI hours you think you need - and we may be able to provide you with that".

It is up to you to decide if you want to contact the support; if not then we should drop all Travis related parts from the repo (and you'd then like to to remove access fro travis to the repo, too).

[GitMensch]

Note: the new URL is https://docs.travis-ci.com/user/migrate/open-source-repository-migration#migrating-a-repository @LADSoft can you try to follow its steps and see if this brings us back? Should be not more than half an hour, possibly much less.

[LADSoft]

@GitMensch, do you know offhand if they relaxed the time limitations?

[GitMensch]

I don't know it, just read about "always free for opensource" and "if the monthly ci-minutes are not enough please get in contact and we see if we can once or permanently provide more".

Of course it would be reasonable to possibly not trigger Travis for each commit, maybe only when a travis branch is pushed which could be done from time to time? Something like this could be tested if we'd see it to be necessary and we still find travis reasonable (if not then it should be removed from the repo completely).

[LADSoft]

yeah when i looked at this they weren't giving out monthly ci-minutes by default, but I figure if we did things at the level we were doing before, if we budget for three builds a day (which is way overkill) we could do with about 500 ci-minutes a month... it would be significantly more if we tried to do windows builds as well... maybe I will talk to someone and see if it is possible...

[LADSoft]

@GitMensch, on this url which I got when I logged into the site, https://app.travis-ci.com/getting_started it indicates the choices are 'free trial' or 'paid'. They want a credit card if you do the free trial. They don't seem to indicate anything about opensource support on that page.

I did find a page about open source: https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com/ but it was written back in 2018 when they were first merging the two sites. Like I've said elsewhere in this issue, I think they've changed course since they wrote that, and are no longer supporting open source. Certainly what they said about migrating the open source project and building it with the allotted minutes didn't work when I tried it; as I recall they weren't giving any minutes whatsoever; I couldn't test the migration and that is why I gave up.

Do you have any other web pages to add that might shed further light on this?