Crossfeed suggestion

[pedromorgan]

Ok so code in various places etc..

Can I propose we move it to https://github.com/freeflightsim/crossfeed

and make that the official version and close down others.. The only reason for github is beacuse.. the keys and stuff are already there vs m2 and also can use jenkins etc for remote builds etc..

? otheres ?? https://github.com/fgx/crossfeed https://github.com/geoffmcl/crossfeed etc

[geoffmcl]

@pedromorgan hi Pete, no the code is not in various places...

For me, it is only in one place -

https://gitlab.com/fgtools/crossfeed

As can be seen, it is a few years since the old fgx/crossfeed was updated... so no idea as to its current state... from my perspective, it could be deleted, to avoid any confusion... if there is any...

As to keys, yes it does require an addition the $HOME/.ssh/config for each repo addressed... sorry about that... add -

Host gitlab.com
 Identityfile /blah/blah/blah

Then, do we agree on the official single source of crossfeed?

And, as always, would love to hear about any bugs, and/or Feature Requests, of crossfeed, against that base source...

Am all ears ;=)) want to progress the app...

Regards, Geoff.

[pedromorgan]

Well gitlab for me is a problem cos I dont wanna other set of credentials and also mainly That I got a relationsip setup between github and m2.server... and also issue tracking etc etc But that is me being lazy

So maybe we come to a cool comproise.. set up mirror to gh.com/ffreeflightsim/crossfeed

Note that I also need to update the docs gen etc etc... which means that simgear, flightgear, fgms etc are already on the system http://api-docs.freeflightsim.org/

Forward Thinking

I'm getting a Fibre here at home soon (bye bye satellitel)

So i want the whole online stuff to run on an RPI3 everm

So I want a package called the "flightgear-server" package and that inclides

1. = fgms (local and remote)
2. = crossfeed
3. = compile farm etc
4. = local map server

But i dream on..

Certainly for ME it would work if there was a pull request soon with and /etc dir containing

• supervisor config (fgms + xfeed)
• fgms config
• crossfeed config
• nginx config
• and logging options for binary dump and replay
• ie a full on fgms playground on a PI

But am thinking out loud.. and an RPI is possible to be a relay server right ??

[pedromorgan]

Ok so looking at this..

https://gitlab.com/fgtools/crossfeed

Can the "csv file pl" script be in that repos..

And I try setup keys to rebuild on M2.. in next few days..

I need to scritpt this using fabric

pete

[geoffmcl]

Hi Pete,

Well gitlab for me is a problem cos ... But that is me being lazy

I really understand, and it is me being very lazy concerning setting up yet another GitHub mirror of crossfeed!

Is that not what github.com/fgx/crossfeed already is?

It could be re-based to my current... why yet another mirror on gh.com/freefightsim? What is the difference? What is the gain? But, really, knock yourself out, if that is what you want...

It seems you are forgetting some history here!

I wrote crossfeed (cf_client)... moved it to fgx... public... had lots of help, ideas, ... from you, and others... but then some things were done that I did not want/like... I do not even remember what now... so I withdrew again to my private, originally gitorious, later gitlab repo... as my playground...

But, yes, I do not mind trying to open it again... with care...

I have not checked exactly, but I do not think github.com/fgx/crossfeed is that far from gitlab.com/fgtools/crossfeed...

Anyway, as stated, do not yet see the gain in adding a 3rd crossfeed repo...

I'm getting a Fibre here at home soon (bye bye satellitel)

That sounds like very good news ;=))

I have what I thought was a very good ISP connection, Orange, ADSL, but see a lot of slow down when I upgraded to Windows 10... not nice!

While I really like my cheap RPI2, and thought recently of buying a new RPI3, I find it very slow at some things...

First it has very limited RAM memory - 1 GB - but in my old mainframe epock, they worked with numbers like 128 KB of fast RAM... so 1 GB is a lot...

Second, the HDD attached is only via USB 2.0... so that also slows down things... pure disk io...

But given time, my RPI2 was able to completely download and compile SG/FG... using -

https://sourceforge.net/p/flightgear/fgmeta/ci/next/tree/download_and_compile.sh

Of course, due to the RAM, and the RPI2 video arrangement was not able to run the result ;=() But it built fine...

It seems to me you should do this on m2, to have an install of SimGear available for other things... like building crossfeed!

This would be a great step forward for m2...

= local map server

You should really check out FGFS Phi - --httpd=8888 - user interface... if you do not already know it...

It already includes a map display... in 2D... tracking your position... can show your track, add navaids, traffic, etc, etc... needs enhancement, additions... defined in fgdata/Phi...

But is there, now... http://[IP|locahost]:8888... easy...

This renders some things offered in FGx Launcher, and even Atlas/Map for that matter, rather redundant!

So to me some dreams are already being realized ;=))

... an RPI is possible to be a relay server right ??

Yes, this seems an easy task for a RPI machine... provided it has a reasonable net connection... does not require much of anything... to just relay IP packets...

Dreams are part of the fun of it...

https://gitlab.com/fgtools/crossfeed Can the "csv file pl" script be in that repos..

Err, no! For me the perl scripts are nothing directly related to the C/C++ crossfeed project...

In fact the cfjsonlog.pl is only possible because of your great DNS - http://crossfeed.freeflightsim.org/flights.json...

So, I am not likely to fold this script, and some others like cfcsvlogs.pl, to build the html stats... into the crossfeed project...

They will remain separate... sorry...

Regards, Geoff.

[theo-armour]

I am enjoying the dialog.

But I do get stressed when you go way off-topic on in my issues.

Please do start your own issues - and go as off-topic there as much as you please.

And let me be able to close my issues when they are resolved.

Thanks in anticipation....

On Tue, Aug 9, 2016 at 10:07 AM, Geoff McLane notifications@github.com wrote:

Hi Pete,

Well gitlab for me is a problem cos ... But that is me being lazy

I really understand, and it is me being very lazy concerning setting up yet another GitHub mirror of crossfeed!

Is that not what github.com/fgx/crossfeed already is?

It could be re-based to my current... why yet another mirror on gh.com/freefightsim? What is the difference? What is the gain? But, really, knock yourself out, if that is what you want...

It seems you are forgetting some history here!

I wrote crossfeed (cf_client)... moved it to fgx... public... had lots of help, ideas, ... from you, and others... but then some things were done that I did not want/like... I do not even remember what now... so I withdrew again to my private, originally gitorious, later gitlab repo... as my playground...

But, yes, I do not mind trying to open it again... with care...

I have not checked exactly, but I do not think github.com/fgx/crossfeed is that far from gitlab.com/fgtools/crossfeed...

Anyway, as stated, do not yet see the gain in adding a 3rd crossfeed repo...

I'm getting a Fibre here at home soon (bye bye satellitel)

That sounds like very good news ;=))

I have what I thought was a very good ISP connection, Orange, ADSL, but see a lot of slow down when I upgraded to Windows 10... not nice!

While I really like my cheap RPI2, and thought recently of buying a new RPI3, I find it very slow at some things...

First it has very limited RAM memory - 1 GB - but in my old mainframe epock, they worked with numbers like 128 KB of fast RAM... so 1 GB is a lot...

Second, the HDD attached is only via USB 2.0... so that also slows down things... pure disk io...

But given time, my RPI2 was able to completely download and compile SG/FG... using -

https://sourceforge.net/p/flightgear/fgmeta/ci/next/ tree/download_and_compile.sh

Of course, due to the RAM, and the RPI2 video arrangement was not able to run the result ;=() But it built fine...

It seems to me you should do this on m2, to have an install of SimGear available for other things... like building crossfeed!

This would be a great step forward for m2...

= local map server

You should really check out FGFS Phi - --httpd=8888 - user interface... if you do not already know it...

It already includes a map display... in 2D... tracking your position... can show your track, add navaids, traffic, etc, etc... needs enhancement, additions... defined in fgdata/Phi...

But is there, now... http://[IP|locahost]:8888... easy...

This renders some things offered in FGx Launcher, and even Atlas/Map for that matter, rather redundant!

So to me some dreams are already being realized ;=))

... an RPI is possible to be a relay server right ??

Yes, this seems an easy task for a RPI machine... provided it has a reasonable net connection... does not require much of anything... to just relay IP packets...

Dreams are part of the fun of it...

https://gitlab.com/fgtools/crossfeed Can the "csv file pl" script be in that repos..

Err, no! For me the perl scripts are nothing directly related to the C/C++ crossfeed project...

In fact the cfjsonlog.pl is only possible because of your great DNS - http://crossfeed.freeflightsim.org/flights.json...

So, I am not likely to fold this script, and some others like cfcsvlogs.pl, to build the html stats... into the crossfeed project...

They will remain separate... sorry...

Regards, Geoff.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/fgx/crossfeed-dailies/issues/2#issuecomment-238621055, or mute the thread https://github.com/notifications/unsubscribe-auth/AAhbKgiQnbbOyd2WmOFXc2DVZ8keYmaZks5qeLPPgaJpZM4JflYb .

[pedromorgan]

@theo-armour sorry, but this is an issue between geoff and myself...

Ok so Gitlab is fine.. and we nuke fgx/crossfeed.git ;-))

and use gitlab/fgtools/crossfeed.. and take it from there..

Re the "logs"//

At the end of the day what I assume we want is...

1) this repos crossfeed-daillies web facing so end users can download eg crossfeed-logs.freeflightsim.org 2) the scripts and cron to generate them...

So ball is in your court... maybe and idea would be that..

1) you use geoff@m2 and checkout to /home/geoff/crossfeed-dailies 2) get your perl "cf-logger" script to write to that directory.. 3) setup supervisor so it starts and restart the "cf-logger" (as geoff user, I;ll setthat up when I know where script is) 4) setup a geoff crontab to git add/commit/push every day.. (means key needs to be setup between gh+m2 on your account) 5) expose ~geoff/crossfeed-dailies to internet at crossfeed-logs.freeflightsim.org

Please tell me what to do ;-))

[theo-armour]

@pedromorgan

Ooops. My big bad!

This issue was opened by you,

Please accept my humble apologies for my wrong rant.

[pedromorgan]

No prbolems... I'm working away on this project in golang http://ags.daffodil.uk.com/widget and its a mix of golang, js and a pain in the "context switch"...

Much like the issues here.. I probably "miff" @geoffmcl a bit more as am playing catchup with current runnings..

BTW.. What subdomain u want these logs on ? XXX.freeflightsim.org.. U decide what XXX is.. and argue with paris ;-)))

[pedromorgan]

@theo-armour U or IRC eg freenode or irc.flightgear.org...

irc.flightgear.org is cool cos like minded people..

[geoffmcl]

Hi Pete,

Re: gitlab.com/fgtools/crossfeed

I had already cloned this into my geoff space -

/home/geoff/fg/crossfeed

I have already setup my gitlab ssh, so could even push from there... in m2...

But as mentioned can not build crossfeed without first building simgear... I will get around to trying this using download-and-compile.sh, in my user space, if you have no objection...

But that is not an important issue, so can wait... we have a working 2014 fg cf_client running fine...

And see no reason to nuke fgx/crossfeed.git ;=))

I will, in the coming days, push a few words to it, making it clear this is an old fork, and giving a clear link to its present address...

Re: crossfeed-dailies

Use geoff@m2... now you are talking ;=))

I can do 99% of this, as geoff, easy, but much, much more difficult, for me, as fg...

Please understand, my main use is windows, so in my user space, geoff, I have created lots of aliases between the dos commands and the bash commands... like dir -> ls, type -> cat, move -> mv, etc, etc, etc... all through .bash_aliases...

Then I create little tools in /home/geoff/bin, which is added to my PATH when I log it, and it contains many little help scripts, like gts, which does git status, navigation commands, like . hcf, which switches me to the crossfeed source, again, etc, etc, etc... lots...

Now what I find is that when I switch to say fg, I lose all that windows/dos help, and then really struggle to remember the correct bash commands... very, very slow and difficult going for me...

Anyway, the script command to run from supervisor is -

/home/geoff/fg/scripts/cfjsonlog.sh -d /home/geoff/fg/crossfeed-dailies

Now the cron command, to run each day after midnight is as you proposed -

DT=`date +"%F"`
cd /home/geoff/fg/crossfeed-dailies
git add *.csv
git commit -a -m "$DT"
git push [origin gh-pages]

I do not think origin gh-pages is required, and note it is gh-paes, not master, in this case...

Now this has the problem, that at that moment in the morning, there would be two CSV files to add... the just completed 30+ KB yesterday CSV, and the new today few K CSV... so that means we would also be adding a partial file to the repo...

That is no problem, in that the commit the next morning would include the now completed yesterday 30+ KB CSV... as modified... and so on...

Now we could get fancy with -

DT=`date --date='yesterday' +"%F"`
FILE="flights-$DT.csv"
cd /home/geoff/fg/crossfeed-dailies
git add $FILE
git commit -a -m "$DT"
git push

That is only adding the yesterday completed file, not the newly created today file... and could have some error checking...

DT=`date --date='yesterday' +"%F"`
FILE="flights-$DT.csv"
cd /home/geoff/fg/crossfeed-dailies
if [ ! -f "$FILE" ]; then
  echo "ERROR: Can NOT find $FILE! Aborting!!!"
  # complain, report somewhere - I **need** attention
  exit 1
fi
git add $FILE
git commit -a -m "$DT"
git push

Or maybe not... as you like, but then users have to be warned that the last file will always be just partial... just minutes old probably...

Re: Exposing XYZ.freeflightsim.org

Well, we should certainly have a discussion about that... What do we want to expose exactly? For what purpose, use...

I personally have not thought this too far forward yet...

And - as previously asked - do we want to generate say a daily html, maybe a monthly html, and a total summary html...

As mentioned, I have a WIP script, cfcsvlogs.pl, which can generate html for one file, or all the files in a directory...

It seems more important to me to expose the html summaries, that could even include download links for the raw CSV file...

And such a lot could be done to improve the html, and begin with say an index.html which describes what is available...

As I understand it, Theo can already directly use the github repo to generate some 3D things from the raw CSV...

Is that correct Theo? Or do we need another http link to the actual CSV...

As I say, seems lots of open questions here...

Items for Pete - 1) setup supervisor to run cfjsonlog.pl 2) setup cron to add, commit, push daily... hopefully with some error checking, and only the last file...

For all - decide definitely, what we want to expose from this raw CSV data... as 3D, as HTML summaries, as whatever...

Regards, Geoff.

[theo-armour]

@pedromorgan

U or IRC eg freenode or irc.flightgear.org... irc.flightgear.org is cool cos like minded people..

I really like IRC and would live to participate in theory, but I find that I can't code and IRC at the same time. ;-(

My priority tends to be on coding. And most of my time is spent on coding

So my IRC time is minimal and mostly devoted to family...

---

@geoffmcl

As I understand it, Theo can already directly use the github repo to generate some 3D things from the raw CSV...

Is that correct Theo? Or do we need another http link to the actual CSV..

I don't know the difference between 'raw' and 'actual' CSV, but if it's CSV with number and accessible via HTTP then the 3D scripts can access the data and show it.

[pedromorgan]

for downloading via http check this http://stackoverflow.com/questions/17341122/link-and-execute-external-javascript-file-hosted-on-github

Main think it does is send proper Content-Type to borwser

[theo-armour]

@pedromorgan

I've been using rawgit for about a year. Works just fine.

The link you sent actually explains things quite well. I learned a bit more from it. Thanks

[geoffmcl]

@pedromorgan hope you get the chance to setup cfjsonlog.pl soon... but understand that you are busy with RL...

Also now note the directory change - should be -

/home/geoff/fg/crossfeed-dailies/csv

Thanks...

[pedromorgan]

Sorry geoff, am kind busy.. But remind me quickly where cfjsonlog.pl is, so I can drop onto supervisor later

/home/geoff/??/cfjsonlog.pl

Sorry to be a pain..

[geoffmcl]

Supervisor command is -

/home/geoff/fg/scripts/cfjsonlog.sh -d /home/geoff/fg/crossfeed-dailies/csv

Hope this helps...

[pedromorgan]

Ok so its running... I think...

[geoffmcl]

@pedromorgan, yes, it seems to be running fine... thanks...

Concerning the second part, setting up a daily cron job to add, commit and push the new csv files to the repo... please hold off on this for a few days...

There is the initial problem of the partial flights-2016-08-14.csv... I will correct this manually to my full 2016-08-14 from here...

Then will write an m2 /home/geoff/bin/updcflogs.sh, and test it, manually, for a few days... until all is in sync...

Then, eventually, I suppose we could run that, adding it to say cron.daily... the only thing I do not exactly understand is how to run the cron job in my user name, geoff...

I think that user name is very necessary to do the repo push... ie use my .ssh authentication... or maybe I do not understand something here?

But, as I say, leave it for a few days... I will revert when I have this next part sync-ed, and running smoothly... thanks...

Re: HTML Information

As you can see, thanks to Theo making it easy ;=)) I now have a few samples in https://fgx.github.io/crossfeed-dailies/index.html

This is an ongoing total summary... I could do it each day, but that seems too much, so much repeated information... at present will try to add that say once a week, or so...

At present we just have about 1.2 months worth of information... when we have more than two months, I can also see adding a Monthly Summary, of some sort... all generated by scripts...

I am also working on generating an svg graphic of daily min/max users... when, and IF, I get through this svg generation learning, that would open the way to generate other graphs...

But, as asked elsewhere, needs some ideas on what else to show?

Theo has already presented a 3D global representation - that's one idea, but... come on guys... masses of information... also need ideas on html presentation... try to make it useful...

[pedromorgan]

@geoffmcl re cron

The way I do it is to create a krontab.txt file in /home/geoff dir

and stick the "config" in there.. (see cat /home/fg/krontab.txt for example)

then load the cron with crontab krontab.txt and list the crontab with crontab -l

Note that the crontab will run with your account permissions..

[geoffmcl]

@pedromorgan, thanks for making the setting up of a geoff cron look easy ;=)) As mentioned, will try that in a few days...

But Pete, you are a web information person... how about some ideas of how to present this information... pages, tables, graphs?, etc, etc...

Remembering, this is github's (mini) gh-pages html server, so we can not get too fancy... but I can gen. it offline, and only push the final, simple html, and javascript, if needed...

Please give it a few minutes thought... thanks...

[pedromorgan]

am kinda "on holiday" from wednessday (my cousin away and aunty sitting she's 93).. so will have time

[pedromorgan]

err... links on README.md dont link to html pages...

[geoffmcl]

Oops, forgot the folder change... should now work... thanks...

[pedromorgan]

I cant figure out what use the pages are..

If it was a website, with postgres behind, then cool, can query stuff, a user and aircraft et all.. which would be more fun

[geoffmcl]

@pedromorgan, wow, yeah, let's put this CSV info into postgres... thanks for the suggestion...

Yes, this CSV data being collected can be put into a postgres database - absolutely! FOR SURE - and put on a real server... m2... the CSV is now already being collected there, daily... quietly... low cost... and will shortly update a repo with this raw data... ie published the raw data...

I could use my cfcsvlogs.pl to populate such a database... write sql to add records... let's muff out some specs on this, and roll...

The CSV header is... records are json blocks... 3 to 54... read every 5 seconds... the update is the UTC time stamp of the record...

fid,callsign,lat,lon,alt_ft,model,spd_kts,hdg,dist_nm,update,tot_secs

But how to record those?

Then that opens the way to say a cflogs.freeflightsim.org?callsign=ME, or cflogs.ffs.org?model=c172p, etc... query, from any site...

Certainly much more fun, and work...

But, without your help, none of this will happen...

So, for now, I can only continue to gather the raw stats... and offer to generate a poor mans static html view of the data... Which probably should be there anyway, as a backup, when the best laid plans fail... and I was really only seeking ideas on that...

As noted, have pushed the next from RPI2/tmp/WIN7-PC/repo, to begin the m2 sync...

[pedromorgan]

We continue this discuss on https://github.com/fgx/crossfeed-dailies/issues/6

[pedromorgan]

http://imgur.com/4F3D5Pn

[geoffmcl]

Just pushed flights-2016-08-15.csv to the repo using /home/geoff/bin/updcflog.sh... had to make some fixes, but seems to now be working fine...

Will conduct one or more manual tests over the coming days... and if all is well will add a cron job to run this each day...

Getting there ;=))