Cool dailies

[pedromorgan]

Few questions from pedro to @geoffmcl @theo-armour Where is this input coming from.. Hopefully its on m2 maybe as xfeed ?

This is what I am after in flight tracker.. cos with postgres, u can kinda read/parse these files.. and stick into postgis as positions and paths...

Hey looking forward to this..

@geoffmcl - we need to automate this preferably with jenkins instance and a couple of slaves... so if one goes down, other one carries on etc..

End of day results and stats etc..

[geoffmcl]

@pedromorgan hi Pete... have manually pushed a few more dailies, added a few words in the readme.md... and pushed a few samples of the html that can be generated...

So as you can see, it is all done manually, starting with a script running in my little RPI2, running 24/7, manually monitored by me... zipped and copied to my tmp, then to other places...

It would certainly be useful if it could be automated, on m2... it seems the script running under supervisor would be sufficient... if it stops, just re-start it...

And then we need a way to copy the daily CSV file to this repo...

Anything you can do to get this happening would be much appreciated... thanks...

[pedromorgan]

so it could be straightforward..

• checkout git to eg /home/fg/csv_logs
• run script (on cron or supervisor) to write file to dir
• use cron to add/commit/push at 1am evry morning UTC

So where is script? how does it work etc

[geoffmcl]

Hi Pete,

Yes, all very simple, but some parts are beyond me ;=))

The perl script is in my repo... so...

 $ cd some/dir
 $ git clone git@github.com:geoffmcl/scripts.git # clone the scripts
 $ cd script/temp # get into the 'temp' dir
 $ md temp-flights # important, script will not create dir yet
 $ cd .. # get back to the scripts
 $ ./cfjsonlog.pl   # and it should be running...

That's it folks ;=))

The cf logs will be written the 'temp/temp-flights' dir, changing output CSV file name each new day... cf fetches are each 5 secs...

The part I do not get is running that cfjsonlog.pl script from within supervisor... it can sometimes break, but usually just needs to be restarted over the following minute, or so...

Probably could find that break, and make it more robust, but in my RPI2 it runs for days and DAYS without a problem...

And also I would have trouble setting up a cron job to transfer the new daily to the repo say each 1-2-... AM... as you suggest...

The name of each CSV file will take the form 'flights-2016-08-06.csv', 'flights-2016-08-07.csv', etc...

Do you need any other information at this time?

Hope you can help get this setup and running... thanks...

Regards, Geoff.

[pedromorgan]

So I checkout this repos on the server

/home/fg/crossfeed-dailies

Is the script on the serer already ? Can it be modified to write to the /home/fg/crossfeed-dailies directory ?

[pedromorgan]

And then I knock up the cron script...

cd /home/fg/crossfeed-dailies
git add *.csv
git commit -a -m "<date here>"
git push origin master

Geoff.. is there a one liner to make the date script same in bash as perl ? (not a bash expert by any stretch of the imagination)

[geoffmcl]

Hi Pete,

Is the script on the serer already ?

It wasn't, but just added it to -

/home/geoff/fg/scripts/cfjsonlog.pl

Can it be modified to write to ... directory ?

Sure, just add -d /home/fg/crossfeed-dailies when running the script...

Of course, one must have the right permissions for this...

... date script same in bash as perl ?

Not that I am a bash expert either, but the following seems to do it...

DATE=`date +%Y-%m-%d`

Regards, Geoff.

[pedromorgan]

Ok so copied over to /home/fg/

When i run I get

fg@s15301371:~$ ./cfjsonlog.pl -d /home/fg/crossfeed-dailies
Can't locate lib_utils.pl in @INC (@INC contains: ./ /etc/perl /usr/local/lib/perl/5.14.2 /usr/local/share/perl/5.14.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.14 /usr/share/perl/5.14 /usr/local/lib/site_perl .) at ./cfjsonlog.pl line 28.

So Which lib do i need ?

Or can this be in the "crossfeed" respository along side the udp/json stuff ?

pete

[geoffmcl]

Hi Pete,

Ok so copied over to /home/fg/

Copied what? Just the one scripts file?

Then ok, it tells you what it is missing -

Can't locate lib_utils.pl ...

So, if you copy that, then it would complain about a missing fg_wsg84.pl...

If you also copied that, then it should work... but might fail!!!

BUT, why this per script copying???

If you want the scripts in the fg user space, then clone the repo there... as fg... simple... it is not big...

$ git clone git@github.com:geoffmcl/scripts.git

And use the scripts in-situ... so it can be updated by a git pull, and bounce the running script... no copying...

Sorry, I was not clear that the scripts are a unit...

Most scripts in there have a local dependency, at least to lib_util.pl, but sometimes others... copying one item out of it will fail!!!

And many use the cloned space for other things, like a log... output to temp... so may also fail... Sorry...

But you should have no trouble running... from anywhere... given that user permissions match...

/dir/of/cloned/scripts/cfjsonlog.pl -d /home/fg/crossfeed-dailies

These scripts, in general, have nothing to do with the crossfeed repo, so no I am not likely to move them there...

And I guess this is because I am a Windows person, but do clone this scripts repo in Ubuntu, to test it there... and it all works for me... using them in-situ...

But for sure, this may not be the usual unix way of doing things... ;=)) too bad...

Regards, Geoff.

[pedromorgan]

Please @geoffmcl

I reslly want all eggs in one basket in this case.. I want crossfeed current at http://crossfeed.freeflightsim.org/

and am even curious which code is running on m2? is it https://gitorious.org/fgtools/crossfeed.git (fetch) or something else..

What I do prefer is having all the "xfeed stuff" in one place along with jobs..

So please can u tell me which is head version running on m2, so I can fork and do things. would be helpful

Sorry 2b short guys..

But am working on a mega project, so diving in and out here..

BUT get real.. I need to get git.. and we gonna get it working..

[geoffmcl]

@pedromorgan hi Pete,

Re: Which crossfeed (cf_client) running on m2?

Well, looking at /etc/supervisor/conf.d/crosffed.conf it seems it is running

command=/home/fg/crossfeed_build/cf_client -c /home/fg/etc/cf/ini

It seems that was built Oct 6 2014, from the source in /home/fg/crossfeed, which is indeed an old gitorious source of around the same date...

When gitorious closed that source was moved to gitlab -

https://gitlab.com/fgtools/crossfeed/

It looks like I did that move Dec 28 2012, and although there have been some 80 commits, but I do not think any were actual code bug fixes... all minor tidying up...

In fact just eyeballing a diff between the gitorious and latest gitlab source seems to confirm this... of course the version has been bumped to 1.0.30... but it seems no major code changes...

Will leave it up to you to decide to leave the current version running - no known bugs in years - or clone and build the latest gitlab source...

But I tried a build of gitlab clone in my space, but it failed...

$ cd /home/geoff/fg/crossfeed/build
$ ./build-me.sh ADDSG ADDMG NOPGSQL

First because I do not know where the libSimGearCore.a has been built!!!

So where has simgear been built? Crossfeed now has a dependence on the core library...

Need it to add a SGDIR=/path/to/sg to the above build-me.sh command...

With this information, I could continue, and check the gitlab crossfeed build works in m2... but need SG... anywhere... somewhere...

Re: Getting cfjsonlog.pl working in m2

Not sure what you want me to do here?

All what eggs, in what basket ;=))

You have to be much more explicit with me...

Sorry 2b short guys..

No problem... as and when you get the time... as always, RL should come first...

Meantime, RPI2/Geoff are continuing their daily stat gathering job ;=))

Regards, Geoff.

[geoffmcl]

Just added last 2 days 06 and 07... time for a git pull... thanks...

[geoffmcl]

And then the latest stats are here -

http://geoffair.org/tmp/20160808.htm

Took 8:32 to cook, using cfcsvlogs.pl... gathered from now 32 CSV dailies available...

Sampled 19,166 flights, flown 277,875,663 nm., probably not accurate due to having no way at present to detect say re-location, or time warping, in some 674 models, by 2,047 callsigns, est. total flight time of 557 days 7:51:47 hh:mm:ss...

Have FUN!

Regards, Geoff.

[theo-armour]

@geoffmcl

Are the .CSV files available on geoffair.net as well? Or just ZIP files?

[geoffmcl]

Just ZIP files... copied from my RPI2 machine (manually) to there...

[geoffmcl]

This discussion seems continued on #2, and expanded, so closing this...