[mickmek] analysing btcusd spread vs orderbook (16h)

[shadiakiki1986]

Hi @MickMek Here's the github issue that we will keep updating to track our progress.

• [x] warm up (~1h)
  • [x] download kraken orderbook on BTCUSD,
  • [x] calculate spread of best bid/ask,
  • [x] plot spread vs volatility
• [x] sample plotting of historical data (~8h)
  • [x] ~download historical dataset of BTCUSD on an example exchange~
    • [x] just download live orderbook snapshot
  • [x] set up notebook that reads the dataset and yields 2 plots: spread timeseries and volatility timeseries
• [x] recording data and plotting the recorded (~7h)
  • [x] ~set up script to download historical dataset every 7 days (since the source you found is rolling over 7 days) (current exercise stores in csv files)~
    • [x] set up script that fetches the live orderbook and stores it in a postgresql database (check example notebook 0.2-shadi-pandas postgresql.ipynb)
      • the database is cryppro_v0 and the table for the top bid/ask is btcprice whereas the full orderbook is in btclobs
  • [x] set up cron job every hour to run the script above
  • [x] set up notebook again that reads the dataset from the database and yields 2 plots: spread timeseries and volatility timeseries
  • [x] set up an alert system on crashes in the market so we can monitor the spread/volatility more closely
    • this is also a cron job, and another script. It just queries the database for latest price for today and first price for today, does if latest - first > 5% then send email
    • [x] test this by setting to 1% and checking that we receive an email

Comment set 1 on the current notebooks

• [x] rename notebooks to have "recording" or "plotting database"
• [x] use better numbers, e.g. 1.0 for recording, 2.0 for plotting
• [x] add to the database recording a new table where we record the complete orderbook
• [x] send the email notification from a gmail account whose credentials I will provide by email
  • mickael ended up using market.eye@gmail.com because he was unable to find which security settings need to be changed to permit using cryppro18@gmail.com
• [x] drop the "timestamp" field from kraken as it's not "unique per orderbook recording" and replace with our own timestamp, generated at the time of the recording (let's stick to UTC)

Comment set 2

• [x] code organization
  • make a folder in your home directory called "1-btcusd orderbook monitor" and put all this in it
  • make folders "notebooks" and "scripts" in "1-btcusd..."
  • move the notebooks into the folder "notebooks" and the "*.py" files into "scripts"
  • make sure to update the cron job to use the new paths
  • clear the output in all notebooks by choosing "cell / output / clear" (in preparation for committing into repository for source control)
• [x] write up a README.md in the folder "1-btcusd orderbook monitor" that describes the files, the database table names, the cron job

[MickMek]

Hi @shadiakiki1986 Updates on project progress:

No available historical dataset on BTCUSD orderbooks online, setted up a script that takes snapshots of the orderbook every minute, and stores the information in a folder as an csv file. It sets up its own database of historical order books, and the rest of the code calculates spread/volatility based on this historical data.

For the alert system, the historical dataset of BTCUSD prices (OHLC) were downloaded, with current (live) cryptocurrencies price. The code runs analysis on cryptocurrencies prices, cryptocurrencies correlations, and correlations to gold and SP500 (to take into account some kind of overall financial market trend). Next step is to figure out how to define a crash in the market, as in the conditions to activate the alert.

All of the codes are now in my folder on the server. Some of them do not compile completely due to several issues faced:

• Unable to install krakenex (Python package) to run complete code on the server
• Unable to find file path on server, in order to direct the code where to download and extract data
• Errors in the results for gold and s&p500 analysis (must revise)

[shadiakiki1986]

Ok. Let's meet tomorrow and change the recording to be into a database instead of files on disk. Also, I install packages on the server globally with sudo. I'll install your requested package and let you know. About file paths, I noticed that the path you used was a full path. Just use a relative path to the notebook. Anyway, you won't need it if we change to using a database

And I don't think crashes in BTCUSD are correlated with s&P. We don't care anyway. We just measure a price change > 5% from the opening

On Sun, Jun 24, 2018, 20:33 MickMek notifications@github.com wrote:

Hi @shadiakiki1986 https://github.com/shadiakiki1986 Updates on project progress:

No available historical dataset on BTCUSD orderbooks online, setted up a script that takes snapshots of the orderbook every minute, and stores the information in a folder as an csv file. It sets up its own database of historical order books, and the rest of the code calculates spread/volatility based on this historical data.

For the alert system, the historical dataset of BTCUSD prices (OHLC) were downloaded, with current (live) cryptocurrencies price. The code runs analysis on cryptocurrencies prices, cryptocurrencies correlations, and correlations to gold and SP500 (to take into account some kind of overall financial market trend). Next step is to figure out how to define a crash in the market, as in the conditions to activate the alert.

All of the codes are now in my folder on the server. Some of them do not compile completely due to several issues faced:

• Unable to install krakenex (Python package) to run complete code on the server

• Unable to find file path on server, in order to direct the code where to download and extract data

• Errors in the results for gold and s&p500 analysis (must revise)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/teamshadi/ssoc18/issues/1#issuecomment-399773314, or mute the thread https://github.com/notifications/unsubscribe-auth/AIAOhLohhQa78sOQYWqPXJ6mEyFyt_Sxks5t_82BgaJpZM4UzeQn .

[MickMek]

Update:

Best_min/max_bid and spread are now stored in postgresql table called btcprice, code in server (0.2-mick) also, spread and volatility database analysis was completed in the second part of the code. I'm having some issues with the crontab, I have tried to run the python script with the command python ./0.2-mick-pd_postgre....py, but it need python3 package installed. Any other way I can run the script?

[shadiakiki1986]

In your command, use "python3 filename.py" instead of "python filename.py"

On Tue, Jun 26, 2018, 10:40 MickMek notifications@github.com wrote:

Update:

Best_min/max_bid and spread are now stored in postgresql table called btcprice, code in server (0.2-mick) also, spread and volatility database analysis was completed in the second part of the code. I'm having some issues with the crontab, I have tried to run the python script with the command python ./0.2-mick-pd_postgre....py, but it need python3 package installed. Any other way I can run the script?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/teamshadi/ssoc18/issues/1#issuecomment-400211583, or mute the thread https://github.com/notifications/unsubscribe-auth/AIAOhGYolAFBle_x3eY_IhuWO5TFnFB5ks5uAeWHgaJpZM4UzeQn .

[MickMek]

I've done that, it still asks for a module named '_tkinter' from the python3-tk package

[shadiakiki1986]

I just looked at your script. You need to erase everything after data.to_sql('btcprice', con=engine, if_exists='append') in 0.2-mick-pd_postgresql.py. The purpose of the scheduled job is just to record the data. No point in "plotting" after that

[shadiakiki1986]

Also, remove your "from matplotlib ..." import line

[MickMek]

Update: The tasks are now completed, with all codes and scripts on the server.

Issues:

• I was unable to send a email with the gmail account provided, as gmail's server wouldn't accept my login from an "unsecured app". I have used another gmail account (credentials are in the script).
• still having some issues with the plotting visualization of spread, with x-axis labels very condensed, and not very readable. However, with the same interval between recordings, we might be able back to indexing with integers

[shadiakiki1986]

@MickMek I added a few points to the steps so we wrap up this project. Let's get them done to close this and move on to quickfix or deepcognition.ai

[MickMek]

Update: email test completed successfully (for 0.1% decrease), as well as code organization and updated crontab jobs. A readme file was also added to the folder.

Edit: noticed error in code, it was alerting for price hikes over 5%, change it to alert for price crashes over 5%

[shadiakiki1986]

@MickMek as discussed, please

• [x] remove passwords from your notebooks so we can commit the files into the repository without any security issues
• [x] re-modify notification script to email us upon a hike or a crash likewise

[MickMek]

UPDATE: @shadiakiki1986 the password have been removed from the notebooks, script notifications modified, and the files are ready to be committed into the repository.

[shadiakiki1986]

OK just committed the code. Thanks for your effort on this project