JSON to CSV data conversion function on a per-stream basis

[phoenix1of1]

Hi, first off, I want to say that the Unicorn Binance Websocket API is really great as it take the mess of the Binance API and structures it in to such a useful tool.

I am currently working on my own little bot and one of the things that became a little annoying was having to convert the raw data in to an exportable and usable CSV format where I can then load the CSV files as Pandas Dataframes for further calculation. The Unicorn_Fy module is awesome for taking the very raw json data and making it more readable but I think an executable function to convert that data further would be time-saving and very useful.

I am a novice Python coder but the following code is currently how I "clean" the raw data in to a usable CSV format:

``def process_message(msg): if msg['e'] == 'error':
print("Error, malfunction") time.sleep(3) print("Terminating process") time.sleep(3) sys.exit() else: Data_Type = msg['e']

    Datetimeindex = msg["E"] / 1000
    Datetimeindex = datetime.fromtimestamp(Datetimeindex).strftime('%Y-%m-%d %H:%M:%S')

    Symbol = msg['s']

    Interval_Type = msg['k']['i']

    Open_Price = msg['k']['o']

    Close_Price = msg['k']['c']

    High_Price = msg['k']['h']

    Low_Price = msg['k']['l']

    Volume = msg['k']['v']

    Quantity = msg['k']['q']

    print("{}, {}, {}, {}, {}, {}, {}, {}, {}, {}".format(Data_Type, Datetimeindex, 
                                                          Symbol, Interval_Type, Open_Price, 
                                                          Close_Price, High_Price, Low_Price, Volume, 
                                                          Quantity), file = "insert file name variable here, flush=True)``

I was wondering if this function could be inserted in to handling the kline streams or at least offer an "enable/disbale" function on a per stream basis via a boolean value which would allow each user to add the json conversion function if they wanted it.

On two side notes to this, the variable tags would need to be amended to offer users the full data-set, I am only interested in nine data points for further analysis.

The second side note is that you have to convert the UNIX timestamp to a proper datetime format as aggregation and resampling of kline data is "buggy" without a correctly formatted datetime to use as an index.

Hopefully this makes sense, I know I am "green" in terms of Python coding but I think the json conversion function would be something that would be really useful.

Either way, thanks for taking the time to think about this and consider it.

Cheers

James

[oliver-zehentleitner]

Hello!

This is the wrong repository ;)

Could you move your post to unicorn_fy: https://github.com/oliver-zehentleitner/unicorn_fy

Thanks and best regards, Oliver

[phoenix1of1]

Apologies Oliver, should have used my common sense there hehe.

Cheers.