My upcoming "rampage"

[IngwiePhoenix]

So... First, hey everyone of the team. I did not know where to put this, but maybe that one here should do.

As of the 18th, I will have a winter break from school. That means that I can dedicate my time to coding - to both my own and SalvationDev's projects. But what will that be? I thought to announce what I am planning first. So here goes:

• Create a branch to restructure the source tree.
  • Binaries in git? That's a strict no-go. These clutter up the git object tree and in general are hard to diff upon... Instead, I will be adding a shell script to download required binaries (namely node-webkit, probably)
• Create a script to bundle launcher AND app into a single .app bundle under Mac OS X.
• Change the YGOPro source tree from using premake to CMake.
  • Irrlicht is NOT EASY to be compiled under OS X. You have to know the tricks and bits to do so. A note or proper fix should be added.
  • Libraries like libevent and other dependencies are not available by default but could properly be noted down either within the README or when CMake fails - aka. better error reporting.
  • Add a "build target" to prepare a release, to download assets, and the like.

I am thus planning further code cleanup on YGOPro and to use FLTK instead of "gframe": https://groups.google.com/forum/#!topic/fltkgeneral/rbrPEo6bQWw

FLTK is a cross-platform window/widget toolkit. Thru it, I plan to re-implement OCGCore with a new GUI. More cross-platform, better structured, simply said: make it a round thing there.

I may come along other/newer plans. Therefore I would simply suggest to see this issue if I add to it. Untill then, you can give me suggestions.

Oh, besides. I am going to prepare my server to be running a continious integration, thus making the same on my Mac Mini homeserver. Both can then push releases to my upcoming company site http://ingwie.io . I am still writing on it and should release a flat version of it soon. (flat = very basic)

Having buidlbots for Linux and mac shoudl be helpful, since most of the team are Windows users. I, however, have access and am daily using Mac, thus having a rather powerful linux box within an OVH DC. ^^ That, in my opinion, should help with "dev releases"...aka, tarballs of per-commit distributions.

I hope you dont mind these plans and feel free to add to it!

Kind regards, Ingwie.

PS: I originally wanted to make the YGOPro/FLTK combo about half a year ago, but never could move myself to do so. Now I have valid reason to move it. :)

[Zayelion]

_>;;# I understand the overkill thats very much are all problems that need to be addressed. Windows doesnt really have something you can pull from to get the binaries like brew or apt-get in current production versions. Our target audiance ar the ages 19 to 23 and they are not generally tech savvy and have limited patience.

FH is already working on redoing the graphics. Hopefully it addresses something with the 400MB of images with templates. We where hoping on making a web version but the YGOPro network protocol is difficult for me to figure out and rerender into plain old HTML. I was considering using a rendering engine but that would slow progress and require I figure out even more stuff in YGOPro's gframe.

We are using Travis.CI and there is another windows specific one we have access to. Our two servers are already hosted on OVH.

Our main concern is social good then the development so thats why we want to push a mac version out as soon as possible because people that only have access to macs are limited to DN mostly.

[IngwiePhoenix]

You can really just create a tiny NSIS-ish installer for Windows. People click those, and you do not need to be tech-savy for them. Usually they are very tiny (2-5mb iirc) so that could do.

Good to know that you are working with Travis. I can see the build processes within my own Travis account (one of my projects builds there also).

The network protocol might be something you'd use a byte-buffer for in JS. If you map out the ocgcore API into JS scope (thru a v8 c++ extension?) then you can actually access it this way. Node-Webkit should probably be able to use C++ extensions. That at least would be my guess.

YGOPro builds for now, its a hassle but it builds for me. I could probably package a tarball with the binary I have, and that works, and upload it to my server to share it with you, if you wanted. Just mind the font issues.

[Zayelion]

We have someone that makes an fairly advanced and professional looking MSI installation that allows repair and upgrade on the windows platform. I think you are talking about a web installer, we can do those to but I've found them more trouble than they are worth but we could explore it in the future.

Currently I am proxying the TCP connection to a web-socket connection via a management server, it comes down as an disjointed byte-stream that has taken a while to figure out what little we have. We have 2 versions of the source in C++ and C# but its been fairly slow due to the way the protocol is written it assumes massive amounts of state and is very inconsistent.

Yes all we want at the moment is a built MacOSX binary so we can deploy to those users They have been waiting patiently for over a year now to be update to the same cards as YGOPro users. We have someone to test and figure out the font issue for us.

[Zayelion]

We've initiated a documentation effort also but school has gotten in the way for the staff that isnt actively coding.

[IngwiePhoenix]

No, I was talking about an installer to install possible dependencies, such as Node-webkit. I do not like web-based installers myself, don't get me wrong there.

Why are you proxying a non-websockets protocol into web-sockets? that...doesn't make sense to me. I am currently preparing a local workflow/environment to possibly have a proper look at the YGOPro protocol.

I wonder, have you read upon the idea of using FLTK as a new UI for YGOPro? I'd be interested to hear your opinion about that.

Currently I am sitting at class, gonna try to press a tarball thru this tight kind of connection... ;)

[IngwiePhoenix]

By the way. About the 400mb of images; if we edited the GUI, we could have it automatically download the missing image off a remote host. Let's say yapi.salvationdevelopment.com offered a RESTful API - in a way - that would send back both images. Then all you have to do is make sure the client is connected to the internet, then it could download the missing images.

[adrianojn]

YGOPro builds for now, its a hassle but it builds for me.

TIP: compile Irrlicht/libevents separately and put the static libraries on bin/release (or bin/debug). Irrlicht has some outdated files (glest.h) and libevents was _./configure_d for Windows.

[IngwiePhoenix]

Not even that… But the current YGOPro source is not optimized imho. The included irrlicht seems a bit outdated and the provided irrKlang has no OS X binaries. I am going to be re-writing the build system some to include pre-build steps to build libevent and irrlicht plus getting the correct irrklang. The current build system blindly assumes everything to be in place, thus assumes that the installed versions match the given development headers within the source tree. As mentioned, Ill be putting my attention there. :)

Am 11.12.2014 um 09:24 schrieb Adriano Soares notifications@github.com:

YGOPro builds for now, its a hassle but it builds for me.

TIP: compile Irrlicht/libevents separately and put the static libraries on bin/release (or bin/debug). Irrlicht has some outdated files (glest.h http://sourceforge.net/p/irrlicht/code/HEAD/tree/trunk/source/Irrlicht/glext.h) and libevents was ./configured for Windows.

— Reply to this email directly or view it on GitHub https://github.com/SalvationDevelopment/YGOPro-Support-System/issues/98#issuecomment-66586464.

[Zayelion]

The use case is many people that play online are not able to install software like ygopro to their computers because they do not own them or have limited technical know how. In an effort to move the player base primarily from the three major sims we are focused on web development. While DN DevPro and Percy offer advanced gameplay they do little in the way of social control like YVD did to keep toxicity to a minimum and provide stable KDE, media and player interaction gates. They are in odd places at the moment and controlled by people that do not seem to mean well.

If you know a stable way of sending normal live bi-directional TCP data to a browser without web sockets I am very open to hearing about it.

In the windows world Node-webkit isnt a dependency its the program itself, its modified with an icon. The next two dependencies are the Pepper Flash plugins included already (and unlikely to update) and there is a delivery system for that and any thing else I wish to push to the clients including 'version-less' YGOPro updates. The only thing update.js can not update is the executable itself which there is an npm package for. So to add additional dependencies I just put them on the server, run update.js and everyone gets the update. If only it ran as fast and smooth as rsync.

While I know git doesnt handle binaries well I have a strict batteries included policy. I've found it helps new and or frustrated staff members and keeps development pace high. This is especially important when working with people that have limited and or specialized skill sets.

As a web developer I dont have an opinion on FLTK. Irrlicht is an nightmare to get working on windows also by the way due to its DirectX dependency. Im not saying reworking the interface is a bad idea, but @Fluorohydride wrote it and its still taking him over a year to reskin it. Only the deck editor works correctly last I heard and he uploads/updates about 2-300 lines of C++ every morning or so. Given that I think its a rather epic and gangrenous task that cant be completed in a few weeks unless this toolkit is a special type of amazing.

[adrianojn]

the provided irrKlang has no OS X binaries

ambiera.com/irrklang

I already researched about a gframe/Irrlicht replacement and the best option is Qt QML. Is the only library able to keep complexity low as it uses JavaScript in many places (but still require C++).

[IngwiePhoenix]

http://ingwie.me/o.o/ygopro.tgz - There. Mac OS X distribution! I just hope I havent missed anything. To see if I did well, youll have to launch it from the console.

tar xvfz ygopro.zip
cd ygopro
./ygopro.app/Contents/MacOS/ygopro

Whatever error you get, let me know so I can update this .tgz . It contains only the ACTUAL game, not the server/client stuff.

[Zayelion]

`>_> We are not assuming technical skill with the people using this. Also MacOSX has a native unzip function for tgz? I know it seems rather silly but we are assuming little to no technical skill from users.

[Zayelion]

Adding to the server and updating to 4.0.0

[IngwiePhoenix]

Yes, Mac will easily unzip (well, untar) the archive. Thing is, I can not verify that external dependencies are all matched! That is, that as soon as you run into issues with it, I need to know to uqickly fix this tarball up. Not all users are plain silly. ;)

[Zayelion]

will do!

[Zayelion]

ok just getting around to inspecting the tarball. It doesnt contain the launcher...

[IngwiePhoenix]

I told you, its -just- the YGOPro game. The launcher, as its made in Node-Webkit, is a different story I need to work on. I have Node-Webkit installed already and am going to see how to properly pack the launcher for OS X.

[Zayelion]

When you do that leave it 'unpacked' ship it open so we can edit and update the files from the server individually. Like ignore the stuff on the node-webkit wikia, all that zipping is not needed. We are taking a calculated risk. And again thank you, everyone has been stressing out over not having a mac release for about a year now. Its a strong part of our community that has been left out mostly.

[IngwiePhoenix]

Just as I had promised, I currently am going over a local branch osx.ingwie. I noticed that NodeWebkit was inside the repository as a duplicate. Having it in there already made the download take forever (currently on a 200kb/s connection) but a duplicate seems a bit overkill.

Then I launched the server as node server and got the notice that it was running - well, a Salvation Dev. slogan apepared and I take it it runs now.

But when I fired up the node-webkit package...

Ingwie@Ingwies-Macbook-Pro.local ~/Work/YGOPro-Support-System/client $ /Applications/node-webkit.app/Contents/MacOS/node-webkit package.nw 
[22122:1220/221225:ERROR:breakpad_mac.mm(238)] Breakpad initializaiton failed
[22123:1220/221226:ERROR:breakpad_mac.mm(238)] Breakpad initializaiton failed
[22121:1220/221226:INFO:CONSOLE(12)] ""/Applications/node-webkit.app/Contents/Frameworks/node-webkit Helper.app/Contents/MacOS/node-webkit Helper"", source: file:///var/folders/kz/l3s2fpd51ln9_p9ymvqbyshm0000gr/T/.org.chromium.Chromium.q45rRM/interface/js/offline-server.js (12)
[22121:1220/221226:INFO:CONSOLE(20)] ""darwin" "./application_mac_ygopro"", source: file:///var/folders/kz/l3s2fpd51ln9_p9ymvqbyshm0000gr/T/.org.chromium.Chromium.q45rRM/interface/js/offline-server.js (20)
[22121:1220/221226:INFO:CONSOLE(787)] "Uncaught TypeError: Bad argument", source: fs.js (787)
[22121:1220/221226:WARNING:simple_index_file.cc(337)] Could not map Simple Index file.

And the view itself: https://www.dropbox.com/s/v3nyx8gfetrljx2/Screenshot%202014-12-20%2022.17.32.png?dl=0

...I can't copy text from that window.

So how do you usually launch node-webkit? Maybe it's just me not launching it correctly.

[Zayelion]

The second copy of node-webkit is going to be for a server side instance for allowing control when people are on the server, it hasnt been worked on at all.

Usually we just start the instance in the client folder.

[IngwiePhoenix]

I did npm install within the root already.

I re-read the log and it appears its missing a local folder. Where do the actual game bundles go? (the „native“ YGOPro clients that is) Maybe I just didnt place mine right.

Am 21.12.2014 um 08:19 schrieb Jamezs Gladney notifications@github.com:

npm install in the top level folder from the command prompt. The second copy of node-webkit is going to be for a server side instance for allowing control when people are on the server.

— Reply to this email directly or view it on GitHub https://github.com/SalvationDevelopment/YGOPro-Support-System/issues/98#issuecomment-67762568.

[Zayelion]

../server/ygocore https://github.com/SalvationDevelopment/YGOCore

[Zayelion]

I caution not to edit the processingIncommingTransmissions.js and server.js because Im currently working on those. Take a look in client\interface\js\configuration.js. Overall logic starts at client\interface\index.html.

[adrianojn]

@IngwiePhoenix

I talked about QML without giving the reasoning... The features I consider important are:

1. Multi-platform - Windows/Linux/Mac is mandatory, Android/iOS/Windows Phone is a bonus
2. Native look-and-feel
3. Skinnable - mandatory if the library doesn't have a native look
4. Animation support

The libs:

• FLTK - (1)
• GTK2 - (1)
• GTK3 - (1) (3)
• Qt - (1) (2) (3)
• QML - (1) (2) (3) (4)
• WxWidgets - (1) (2)

Another option would be OpenGL/WebGL but will be harder than a GUI framework.

[Zayelion]

What is FH currently rewriting YGOPro2 with?

[adrianojn]

He started with WxWidgets but changed to OpenGL.

[Zayelion]

He's done alot of mergebacks recently, you know if its functional?

[adrianojn]

I don't know. I'll try to compile today or tomorrow.

[IngwiePhoenix]

If he goes with OpenGL, then this is a good step forwards, because it is easier than using Irrlicht.

FLTK may not provide a 100% look and feel, however, the game's frame is completely filled with graphical contents, so you wont even notice that it lacks some nativeness.

WebGL is also an idea - there actualyl are frameworks out there to support it. But it would be quite a bit of work, since you would have to:

• teach Node-Webkit how to work the entire YGOPro NAPI (NetworkAPI)
• map libocgcore (libduel, libfield, libcard, ...) into a v8 API. It is not hard, but you would embed a scripting language into a scripting language - lua into v8.
• Write ontop of a good WebGL framework and have quite some skills in it.
• Hope that Node-Webkit doesnt derp the f- out. When AppJS and NodeWebkit were new kids on the block, they both lacked mandatory support for WebGL. I am not aware of any fact that this has been fixed for Node-Webkit. AppJS is deprecated by now.

[IngwiePhoenix]

I forgot to mention:

FLTK can be skinned and does support animations (GL widget with a gif rendered inside, for example).

Porting to WP, IOS and Android is a thing for itself and is not only achieved trough a GUI framework. For instance, iOS puts a hammerlock on your network traffic, and you need a good amount of ObjC to have it behave right.

[Zayelion]

• The STOC (Server to Client), part of the YGOPro NAPI is mostly mapped, you can see that here: The CTOS (Client to Server) code would be hard but I think I could work it out of the visuals where taken care of.
• The server sends all the visual information, the OCGCore is bypassed mostly when dueling online. Its only needed for replays and offline LAN mode. What little work it does do can be recreated with a JSON Database.
• This is the hard part, taking FH's OpenGL code running it through Emscripten and then wrapping it with WebGL. We would have to create an API to interface to what ever framework was chosen from the network interface to the GUI. Logic for gameplay is pretty simple after that.
• Node-webkit is just a choice in this project because it can launch manage and edit YGOPro without hindrance. If we port to WebGL then there would be no reason to really keep WebGL. Node-webkit also makes bounds every few months so its likely fixed with new chromium implementations, its not soemthing Im worried about. -The final option is finishing the HTML/CSS rendering, it works on all platforms being just HTML and CSS.

[Zayelion]

If anything I think talking to @Fluorohydride and his active team members like @DailyShana about it would be a good start.

[IngwiePhoenix]

ocgcore drives the whole game O.o In fact, it receives all the commands to move cards and triggers callbacks to have the visuals update. I have taken a look into the source - its ultra tiny actually, and showcases how little is needed for it to fully function.

I dont know how many of our team has C++ understanding, but I would offer myself to go and have a talk with them, discuss features they are planning and see what information I can obtain. That is...if FH speaks english.

[Zayelion]

int DuelClient::ClientAnalyze is the function I recreated. It doesnt strike me as processing any Lua. It just looks like un-marshaling a very optimized API. It isnt particular consistent either, which is annoying.

FH speaks english he just doesnt speak alot.

[IngwiePhoenix]

So I went and cloned the ygopro2 branch.

Fazit: Dont. Do. It.

After obtaining all of the OpenGL s-... that is out there, I had to even go out of my way and edit things around and add linker flags...it took me a while to get them into place:

-framework Foundation -framework CoreFoundation -framework Cocoa -framework IOKit -framework OpenGL -lobjc

And yknow what?

Ingwie@Ingwies-Macbook-Pro.local ~/Work/ygopro2/out $ time ./bin/netc 
Segmentation fault: 11

real    0m0.581s
user    0m0.016s
sys 0m0.100s
Ingwie@Ingwies-Macbook-Pro.local ~/Work/ygopro2/out $ time ./bin/nets

real    0m0.008s
user    0m0.004s
sys 0m0.003s

Nothing. These are the only two binaries. But the fact that theyre both called net[s|c] gives a rather nice hint. It appears that he is working on the NAPI. Maybe he finally ovvers a libyp_net?

Now, to ocgcore and how it controls the GUI. In order to also for myself learn a little bit more about how this stuff works, I began to disassemble it from head to toe. I wanted to start with the function you had mentioned, but:

Ingwie@Ingwies-Macbook-Pro.local ~/Work/ygopro/ocgcore $ grep -r "DuelClient" .
<nothing.>

Where'd you see it? o.o

But anyway, I went a bit further and looked a bit more direct... Here is what I found:

There are five base libraries:

$ ls -1 lib*
libcard.cpp
libdebug.cpp
libduel.cpp
libeffect.cpp
libgroup.cpp

and each one is associated with its own task, obviously. They all are reachable by their respective APIs, but the entire "core" is to be met via ocgapi.h. This file can be traced back to:

Ingwie@Ingwies-Macbook-Pro.local ~/Work/ygopro/gframe $ grep -r "ocgapi.h" .
./config.h:#include "../ocgcore/ocgapi.h"
./replay.cpp:#include "../ocgcore/ocgapi.h"
./single_duel.cpp:#include "../ocgcore/ocgapi.h"
./tag_duel.cpp:#include "../ocgcore/ocgapi.h"

gframe is pretty much Graphics frame. It handles the incoorperation between IrrLicht/IrrKlang and registers its callbacks back into the ocgcore.

I will not repost all sorts of code now, but here is what I have understood from the way this architecture and code was done. I did NOT read it all forth and back, because, heeeeell no. I dont even speak Irrlicht. ;)

• ocgcore is initialized together with parameters to find the scripts/ folder - aka. the cards.
• gframe displays the user interface and handles the configuration of the files, mainly letting the user pick the game and alike. Because aside form displaying fancy graphics, it also makes it obvious that the g could mean Game too. For instance, it holds client_card.cpp - and other cards alike - which entail how the callbacks are used together with the network protocol.
• During the game, the callback is parsed and the correct ocgcore API is called to drive the events within the field. speaking of events - this is done using ´libevent`.
• ocgcore responds by triggering callbacks from its side. Lets say the netserver told the client Your opponent tributed monster in zone 2 (not the real thing, but i suppose you see what i mean), then the respective API - let it be Field::moveCard(FIELD::OPP,ZONE::MON,2, FIELD::GRAVE) - is called. This in terms backfires a callback to move the graphical object (the card) across the screen too.

Interestingly, OCGCore is a wonderful example of an embeddable API. You can compile it totally without gframe - which makes the game...

Ingwie@Ingwies-Macbook-Pro.local ~/Work/ygopro/ocgcore $ cloc .
      29 text files.
      29 unique files.                              
       0 files ignored.

http://cloc.sourceforge.net v 1.62  T=0.28 s (101.9 files/s, 93603.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C++                             17             55            260          23621
C/C++ Header                    11            123             99           2463
Lua                              1              1              0              9
-------------------------------------------------------------------------------
SUM:                            29            179            359          26093
-------------------------------------------------------------------------------

...really tiny. It even is less of a size than my personal IceTea project. (Now if you show that to Konami...and look at how "big" their game's core probably is... XD)

The only requirement ocgcore has is a C++11 capable compiler.

Now, what does all my babbling mean?

...that you can actually implement YGOPro completely in a web interface. You do JavaScript and NodeJS, then you probably are well aware of the all-existing event loop. Now imagine, that all the OCGCore API is bridged into JavaScript. Despite the language-to-language embedding, you would be able to draw the field and such using HTML5 canvas, use XHRs to fetch images and card lores.

Put together with your "reverse engeneering" (i just like calling it this, sine youre doing a lot of amazing work into working out the NAPI in JS! kudos there bud) this could actually be a thing.

Okay, I think I am done throwing techno facts around. I just really felt like sharing my findings and alike, I hope I did not scare anyone or alike. And excuse my language... its 3.49AM at the morning. I have had barely any sleep, I am really hungry and I have coded all the last dozen days straight @.@

[Zayelion]

Nothing I havent seen been told or realized myself, I read over gframe and OCGCore to. I dont know C++ so it didnt sink in like that. The mentioned function is in master not ygopro2 in gframe's duelclient.cpp.

OCGCore compiles down on its own and can be used for seperate computing. This and things similar are what run on the major sim servers. Emscripten compiled Lua to JS but Im not sure how to utilize it in that form. NodeJS can connect directly to DLL using node-ffi I started reimplimenting Buttys ygocore as we call it into JS but,... priorities to just get the sim up and running.

That video uses XHR to get its images from my server. But like I said from trying all this a client is just answering questions and not really using its ocgcore, just the serversides one, which doesnt need graphics.

[adrianojn]

@IngwiePhoenix netc needs to be in the same folder of common.xml

I compiled after editing some files but netc does nothing (no error message). Maybe it lacks some config.

[Zayelion]

Winter break is almost over.

• [x] The NAPI is in a seperate git now.
• [ ] We still dont have a deploy for Mac.
• [x] The mac application is deployed in the updater
• [x] A number of binaries have been removed from the git, specifically the server folder. I dont feel the server and client should be split into seperate gits at this time because the client requires the server to be developed on locally. This will become increasingly true as new features are added and the system as a whole makes the crawl to the web. [x] The only binaries are pictures used in the GUI, and 1 launcher and required binaries for each platform.