Create a Getting Started Guide

[eabase]

Hi Ariel,

It would be extremely helpful if you could create a basic Getting Started Guide for how to get something out of this repo. As it stands we need to find our own data feeds (or mock-up feeds) and feed that into this tool. But how to get there?

I think an example from some free or delayed demo feed (or perhaps a small demo DB with pre-populated data) would be very helpful in this case. (What were you using yourself?)

Could you write down the steps needed from cloning this repo to getting the plots/images shown in the README?

Thanks in advance,

[silahian]

Agree One thing I have in mind is to improve the documentation

Anyways, to get you started, you can follow https://github.com/silahian/VisualHFT#how-to-install-and-run-the-project And, to have a quick run and see real-time market data, you must run the demo trading system first (from https://github.com/silahian/VisualHFT/tree/master/demoTradingCore) That system will feed VisualHFT with market data...

I will keep updating that demo so more and more data can be sent

[eabase]

Hi!

Anyways, to get you started, you can follow https://github.com/silahian/VisualHFT#how-to-install-and-run-the-project "The server app must send the following websocket messages..."

Yes, of course that is the first thing I was looking at and trying to understand what you meant. But where is this "server app" (or what server apps are available to use that are compatible)? Where can I obtain a MVP/PoC server app that I could use here? Are there any common trading platforms that can provide this data? (E.g. IBKR, MT5, Webull, Quantower, SierraCharts etc.?)

And, to have a quick run and see real-time market data, you must run the demo trading system first ... "the information must be read from the database (MS Sql Server database)..."

Basically what are the requirement products & software needed to compile and run this?

• We need to install MS SQL? (Can we use other DB's, like MySQL, SQLite3 etc.?)
• We need what development framework for compiling this? (What compiler options, .NET versions, etc?)
  • You mention WPF but there it ends...
• We need what additional data sources to populate the DB?
• Where are the configuration files and how to understand them?
• SW Dependencies?

---

So what are the options and where can I find instructions to implement these?

You mentioned in the README that you were using Binance in the example.

market data is coming from Binance (btc/usd)

That sound like a good idea, so how do you implement this to serve up the data and populate the DB from Binance?

---

Installation Steps

• Step-1

  git clone https://github.com/silahian/VisualHFT

Then what?

[silahian]

Hi!

Anyways, to get you started, you can follow https://github.com/silahian/VisualHFT#how-to-install-and-run-the-project "The server app must send the following websocket messages..."

Yes, of course that is the first thing I was looking at and trying to understand what you meant. But where is this "server app" (or what server apps are available to use that are compatible)? Where can I obtain a MVP/PoC server app that I could use here? Are there any common trading platforms that can provide this data? (E.g. IBKR, MT5, Webull, Quantower, SierraCharts etc.?)

And, to have a quick run and see real-time market data, you must run the demo trading system first ... "the information must be read from the database (MS Sql Server database)..."

Basically what are the requirement products & software needed to compile and run this?

• We need to install MS SQL? (Can we use other DB's, like MySQL, SQLite3 etc.?)

• We need what development framework for compiling this? (What compiler options, .NET versions, etc?)

  • You mention WPF but there it ends...
• We need what additional data sources to populate the DB?
• Where are the configuration files and how to understand them?
• SW Dependencies?

So what are the options and where can I find instructions to implement these?

You mentioned in the README that you were using Binance in the example.

market data is coming from Binance (btc/usd)

That sound like a good idea, so how do you implement this to serve up the data and populate the DB from Binance?

Installation Steps

• Step-1

git clone https://github.com/silahian/VisualHFT

Then what?

Thanks for the suggestion... I always looking to improve the documentation I will be working on it

[eabase]

@silahian

I will be working on it

If you can help me getting setup and point me in the right direction for getting this to work, I can help you with both the documentation and maintenance of this repo, including some of the improvements you mentioned in the README,

[silahian]

@silahian

I will be working on it

If you can help me getting setup and point me in the right direction for getting this to work, I can help you with both the documentation and maintenance of this repo, including some of the improvements you mentioned in the README,

sounds good :)

pre-requisites: 1) download the Microsoft .net framework 4.8 package 2) SQL Server (any version) 2.1) create "hft" database 2.2) run SQL script provided in https://github.com/silahian/VisualHFT/blob/master/SQL%20scripts/table%20creation.sql

The steps should be: 1) clone the whole project 2) open the solution (VS2019 or newer) 2.1) Do a "rebuild all" so you can restore missing dependencies from nuget 3) check missing dependencies 3.1) Telerik lib: there is a folder "Telerik" where you can find all these dll's. Make sure you copy these into your /bin folder 3.2) All the other dependencies are available in nuget, so shouldn't be any issue 4) make sure you can compile the entire solution 5) Run the "demoTradingCore" project. This will gather real-time market from Binance and Coinbase 6) Run the "VisualHFT" project. This will receive the data from "demoTradingCore" and it will display it

I think I didn't forget any step, but let me know if you run into any issues Probably the most difficult part will be to have all the dependencies right, Let me know how it goes

[eabase]

@silahian I am not able to build because I first get this error, for missing App.config file.

App.config : error MSB3249: Application Configuration file "App.config" is invalid. Could not find file

Assuming you wanted the file you called _App.config, to be this one, I simply renamed it.

Trying to build again leads to the following (several) errors:

... VisualHFT\Helpers\HelperPosition.cs(13,7): error CS0246: The type or namespace name 'MahApps' could not be found (are you missing a using directive or an assembly reference?) ...
... VisualHFT\Model\hftModel.Context1.cs(29,24): error CS0246: The type or namespace name 'DbSet<>' could not be found (are you missing a using directive or an assembly reference?) ...
...

The following seem missing or not found:

'MahApps'
'log4net'
'Newtonsoft'
'Newtonsoft'
'WebSocket4Net'
'LumenWorks'
'Expression'
'Entity'
'Newtonsoft'
'Newtonsoft'
'Newtonsoft'
'MessageReceivedEventArgs'
'SuperSocket'
'WebSocket'
'JsonConverter'
'JsonReader'
'JsonSerializer'
'JsonWriter'
'JsonSerializer'
'Entity'
'Entity'
'DbContext'
'DbModelBuilder'
'DbSet<>'

[eabase]

• Also there is only a packages.config for the demoTradingCore directory, where are the packages for the main one?
• Where and what is in your nuget.config?

[silahian]

ok, updated the steps based on my testing on a clean machine Unfortunately, it needs to have the SQL server database installed (updated on the steps below) This is something I want to change, having the database should not be mandatory.... but for now it is :(

pre-requisites: 1) download the Microsoft .net framework 4.8 package 2) SQL Server (any version) 2.1) create "hft" database 2.2) run SQL script provided in https://github.com/silahian/VisualHFT/blob/master/SQL%20scripts/table%20creation.sql

The steps should be: 1) clone the whole project 2) open the solution (VS2019 or newer) 2.1) Do a "rebuild all" so you can restore missing dependencies from nuget 3) check missing dependencies 3.1) Telerik lib: there is a folder "Telerik" where you can find all these dll's. Make sure you copy these into your /bin folder 3.2) All the other dependencies are available in nuget, so shouldn't be any issue 4) make sure you can compile the entire solution 5) Run the "demoTradingCore" project. This will gather real-time market from Binance and Coinbase 6) Run the "VisualHFT" project. This will receive the data from "demoTradingCore" and it will display it

[silahian]

forgot to mention, try to get the latest.. since I updated the config file

[eabase]

@silahian Yeah, I finally got the "demoTradingCore" compiled and running, but I can't get the "VisualHFT" main part to compile. It gives that long list of "missing" (or not found) dependencies, shown above. So it's missing something.

It seem that .\demoTradingCore\packages.config is only used for that directory, so we have no .\packages.config for the root directory of VisualHFT...

Now take a deep breath. I'm doing all this from command line in order to know (and see) what's going on. So it's a small miracle that I got this far! :wink: (As VS2019 is quite bloated and should not be required for running it, I prefer to use VS Code, until more development is needed.)

3.1) ... Make sure you copy these into your /bin folder

What /bin folder? Please be more specific as there are several!

All the other dependencies are available in nuget, so shouldn't be any issue

That's kind of vague. Well, they need (or should) get installed automatically, so how was this actually done?
Did you install them manually, or using nuget install, or dotnet nuget install?

BTW, does the SQL Server provide anything that is needed for the compilation to complete?

[eabase]

SIDE NOTE

SQL Server (any version)

I haven't installed it yet, and have also started converting your SQL script from MS SQL to an SQLite3 DB. The idea being that SQLite3 is super light weight, and for any desktop developments and tests it should be more than sufficient, if not also faster.
Anyway, this would require either a bridge from the DB connector, or a different one altogether, so this would be a future enhancement or alternative and experimental development.

UPDATE: It turned out there is already a super light MS SQL Server Express. I have documented how to use that in another issue.

[silahian]

BTW, does the SQL Server provide anything that is needed for the compilation to complete?

No, compilation should go ok, even if you don't have SQL server... however, it will throw errors when executing it

I'm doing all this from command line in order to know (and see) what's going on.

Now make sense why you are having all these issues... Let's see if we can make it work. That's going to be hard... very... Visual Studio solves lot of these issues... Since the project has reference to its dependencies, already knows where to look for them (and download it if you don't have them locally). Best thing I can do here is to list manually the dependencies needed DigitalRuby.ExchangeSharp.1.0.3 log4net.2.0.8 Microsoft.AspNet.SignalR.Client.2.4.3 Microsoft.Bcl.AsyncInterfaces.6.0.0 Newtonsoft.Json.13.0.1 NLog.5.0.4 PacketDotNet.0.19.2 PacketDotNet.Standard.0.16.0 QuickFix.Net.1.8.0 SharpPcap.4.5.0 SocketIOClient.3.0.6 System.Buffers.4.5.1 System.Collections.4.3.0 System.Collections.Concurrent.4.3.0 System.Configuration.ConfigurationManager.6.0.1 System.Memory.4.5.5 System.Numerics.Vectors.4.5.0 System.Reactive.5.0.0 System.Runtime.CompilerServices.Unsafe.6.0.0 System.Security.AccessControl.6.0.0 System.Security.Permissions.6.0.0 System.Security.Principal.Windows.5.0.0 System.Text.Encodings.Web.6.0.0 System.Text.Json.6.0.6 System.Threading.Tasks.Extensions.4.5.4 System.ValueTuple.4.5.0 WatsonWebsocket.3.0.1

What /bin folder? Please be more specific as there are several!

You have to copy Telerik DLLs into \VisualHFT\bin\Debug (assuming you are running in debug mode)

.... started converting your SQL script from MS SQL to an SQLite3 DB

I am not 100% sure if that would work. Unfortunately, this project is heavenly dependent on SQL server. Very bad thing I know, and something I should address in the near future. Just so you know, you can download the MS SQL Server Developer edition for free... it is very heavy I know, but it will be a better starting point.

[eabase]

🔼 You need to put a newline before the quotation character >, and after the last line of text, otherwise the markups puts it all within the same paragraph.

[silahian]

@eabase wondering if all this information helped you to get the project up and running. Let me know

[eabase]

@silahian

Hi! Sorry for delayed response. Had some friends visiting and also been busy trying make some progress on a more light setup, not requiring a full MS SQL Server, and lighter VS setup, and managed to get it compiled using VS after having installed a shitload of VS extensions and mostly GB of additional compiler dependencies...

Result, are as following:

• Compiles using VS 2019
• Does not compile from command line because of missing dependencies, not found even after full VS install.
  VS do find the dependencies, so my suspicion is that one or more of the following files are missing the required package specifications (shown in earlier comment):

  ./VisualHFT.csproj
  ./App.config
  ./demoTradingCore/App.config
  ./demoTradingCore/packages.config
  ./demoTradingCore/demoTradingCore.csproj

Shouldn't there be a packages.config also in the root directory?

• There are a lot files that seem to have unused imports (Using).
  Maybe remove these... unless they are actually needed elsewhere.

• The dependency System.Windows.Interactivity.WPF is deprecated in favor of Microsoft.Xaml.Behaviors.Wpf.
  Perhaps you can update this accordingly?

• I managed to get this to use MS SQL Server 2019 Express Edition using a LocalDB install that uses 58 MB instead of the mega bloated 6.7+ GB full server install, and can be configured using sqlcmd. I am currently documenting the entire installation process. The server can then be accessed via a named pipe or directly on the LocalDB file (*.mdf).

• Anyway, after running it, it seem that the fields are not populated correctly.

Not sure, but I might need to adjust some of the App.config settings.

  <connectionStrings>
    <!--LOCAL-->
    <add name="HFTEntities" connectionString="metadata=res://*/Model.hftModel.csdl|res://*/Model.hftModel.ssdl|res://*/Model.hftModel.msl;provider=System.Data.SqlClient;provider connection string=&quot;Data Source=(LocalDB)\hft;AttachDbFilename=C:\Users\xxx\hft.mdf;initial catalog=HFT;integrated security=True;MultipleActiveResultSets=True;App=EntityFramework&quot;" providerName="System.Data.EntityClient" />
  </connectionStrings>
  <appSettings>
    <!--LOCAL   -->
    <!--add key="ConnectionString" value="Data Source=(local);Initial Catalog=HFT;Trusted_Connection=True" /-->
    <add key="ConnectionString" value="Data Source=(LocalDB)\hft;AttachDbFilename=C:\Users\xxx\hft.mdf;Integrated Security=True;Connect Timeout=30;Initial Catalog=HFT;Trusted_Connection=True" />
    <add key="RestFullConnection" value="http://localhost:6800/" />
    <add key="WSorderBook" value="ws://localhost:6900/" />
    <add key="ClientSettingsProvider.ServiceUri" value="" />
  </appSettings>
  <entityFramework>
    <defaultConnectionFactory type="System.Data.Entity.Infrastructure.LocalDbConnectionFactory, EntityFramework">
      <parameters>
        <parameter value="mssqllocaldb" />
      </parameters>
    </defaultConnectionFactory>
    <providers>
      <provider invariantName="System.Data.SqlClient" type="System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer" />
    </providers>
  </entityFramework>

It would be helpful if you can tell me what the RestFullConnection and WSorderBook connections are accessing. Do I need to point these to the DB or where are the ports determined?

[eabase]

BTW. In the connection string:

<add name="HFTEntities" connectionString="metadata=res://*/Model.hftModel.csdl|res://*/Model.hftModel.ssdl|res://*/Model.hftModel.msl;provider=System.Data.SqlClient;provider connection string=&quot;data source=.;initial catalog=HFT;integrated security=True;MultipleActiveResultSets=True;App=EntityFramework&quot;" providerName="System.Data.EntityClient"/>

You have stuff that reads:

metadata=res://*/Model.hftModel.csdl|res://*/Model.hftModel.ssdl|res://*/Model.hftModel.msl;

What does that do?

[eabase]

I got it to work! ✨ Are there any key-bindings to the UI?

[silahian]

well done my friend :) I will be also pushing some new updates I've been working on to make the real-time data consumption part more effective.

What do you mean by "key-bindings to the UI"?

[silahian]

What does that do?

The system uses Entity Framework to map the database... so what you highlighted is the connection string for EF

[eabase]

well done my friend :)

Thanks.

I will be also pushing some new updates I've been working on to make the real-time data consumption part more effective.

I suggest you have a look at (and consider using) the MS SQL LocalDB install, unless you have a greater need for a huge DB server, as i have detailed here:

• https://github.com/silahian/VisualHFT/issues/6

What do you mean by "key-bindings to the UI"?

Maybe there should be a way to interact with the app page, opening mock trades, and navigating to the other stats pages etc. (Ok, sure its just a demo.)

I have some problem with the layout, as my screen is not as big as defaulted in the App. Would have been great to be able to move the sub windows around...

[eabase]

@silahian

I will be also pushing some new updates I've been working on to make the real-time data consumption part more effective.

Hi! Any updates?

Wondering if it would it be possible to keep the OB horizontal scale fixed for maybe 1-2 seconds, as it keep jumping around too fast to be useful?

In addition the OBL red/green blocks are very "blocky", making me think they are too "narrow" and only shows a few price levels and thus both jumpy and blocky. Would have been great to see the whole thing a lot wider with many more levels. Or have I miss understood something here?

[eabase]

Some other questions:

1. How to setup & use Strategies?
   (What goes in the JSON file?)
2. How to connect to our own broker datafeed, or using our own API key to Binance/Coinbase?

[silahian]

@silahian

I will be also pushing some new updates I've been working on to make the real-time data consumption part more effective.

Hi! Any updates?

Wondering if it would it be possible to keep the OB horizontal scale fixed for maybe 1-2 seconds, as it keep jumping around too fast to be useful?

In addition the OBL red/green blocks are very "blocky", making me think they are too "narrow" and only shows a few price levels and thus both jumpy and blocky. Would have been great to see the whole thing a lot wider with many more levels. Or have I miss understood something here?

Those charts should be improved I agree… You are right that they are narrow because of the depth levels received from the source. To change that, you should check the "demoTradingCore" project. From there, we are getting the market data and currently getting 5 levels of depth.

[silahian]

Some other questions:

1. How to setup & use Strategies? (What goes in the JSON file?)
2. How to connect to our own broker datafeed, or using our own API key to Binance/Coinbase?

1) I will update the “Strategy” json soon… in the meantime you can check in /Models what it is expecting. Basically are params to monitor how the strategy is doing 2) if you want to replace your own broker feed (crypto or any other market) you just need to look at what “ the "demoTradingCore" project is doing

[eabase]

Any new updates to the strategy?

[silahian]

Any new updates to the strategy?

I just added a demo strategy. However, a lot of refactoring needs to be done on how market data is being consumed by the UI

[drusteeby]

However, a lot of refactoring needs to be done on how market data is being consumed by the UI

If you wanna open another issue with what classes you think need refactoring I can take a look

[silahian]

However, a lot of refactoring needs to be done on how market data is being consumed by the UI

If you wanna open another issue with what classes you think need refactoring I can take a look

This is the issue: https://github.com/silahian/VisualHFT/issues/4

[silahian]

Closing these guidelines since they have become obsolete with the latest changes.