Using devcontainer for debugging the strats and development?

[Jarrodsz]

Since running on an Mac M ARM processor installing packages has been a real issue, also installing freqtrade native is near impossible.

So I started investigaging other options ( still dont like visual studio code) but it provides a devcontainer extension which lets you install all deps somewhat more easily.

In the root of project ./devcontainer/.devcontainer.json contains:

{
  "name": "Freqtrade development container",
  "build": {
    "context": "..",
    "dockerfile": "../Dockerfile"
  },
  "mounts": [
    "source=/YOUR_PATH_TO_FREQTRADE/freqtrade/user_data,target=/freqtrade/user_data,type=bind"
  ],
  "runArgs": [
    "--name=freqtrade"
  ],
  "settings": {
    "terminal.integrated.shell.linux": "/bin/bash"
  },
  "postStartCommand": "/freqtrade/user_data/start.sh"
}

In the root of project ./devcontainer/Dockerfile contains:

I used this for now as it works its a WIP and still not happy with it, but seems most deps are solved:

FROM freqtradeorg/freqtrade:develop

USER root

RUN apt-get update \
    && apt-get -y install zsh \
    && apt-get clean

RUN chsh -s $(which zsh) ftuser

# Install dependencies
COPY requirements-dev.txt /freqtrade/

RUN apt-get update \
    && apt-get -y install git mercurial sudo vim build-essential \
    && apt-get clean \
    && mkdir -p /home/ftuser/.vscode-server /home/ftuser/.vscode-server-insiders /home/ftuser/commandhistory \
    && echo "export PROMPT_COMMAND='history -a'" >> /home/ftuser/.bashrc \
    && echo "export HISTFILE=~/commandhistory/.bash_history" >> /home/ftuser/.bashrc \
    && chown ftuser:ftuser -R /home/ftuser/.local/ \
    && chown ftuser: -R /home/ftuser/

USER ftuser

RUN pip install --upgrade pip

# CUSTOM DEPS
#RUN curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-aarch64.sh
#RUN sh Miniconda3-latest-Linux-aarch64.sh -b -p
#RUN pip install tensorflow-metal
#RUN conda install -c conda-forge -y pandas jupyter
#RUN conda install pytorch torchvision -c pytorch

RUN pip install tqdm quantstats
RUN pip install finta
RUN pip install prettytable
RUN pip install PyWavelets
RUN pip install simdkalman
RUN pip install pykalman
RUN pip install scipy
RUN pip install scikit-learn
RUN pip install xgboost

#RUN conda install -c apple tensorflow-deps
#RUN conda install -c conda-forge -y pandas jupyter
#RUN conda install pytorch torchvision -c pytorch

RUN pip install tensorflow
RUN pip install numpy

#RUN pip install tensorflow-metal
RUN pip install keras
RUN pip install darts

# Start the proxy
#CMD ["bash", "-c", "/freqtrade/user_data/cli/binance-proxy --disable-fake-candles"]
#CMD ["bash", "-c", "/freqtrade/user_data/start.sh"]

# Empty the ENTRYPOINT to allow all commands
ENTRYPOINT []

Would it be an idea to have a community effort to build a solid devcontainer for developing and debugging the strats?

some things i want to extend this with:

1. Use ZSH instead of Bash
2. Be able to figure out how to use another IDE ( like intellij which i personally prefer used it for 10 years ) to set debugger statements in strat code and be able to debug it. this would still require vscode to run and build the devcontainer though.
3. Fix the binance proxy this seems to be somewhat complicated with the need to map it havent dived into it yet. I use the binary file from the repro inside user_data/cli for now which is not working thus disabled. Perhaps not required if working on only 1 or 2 strats.

There is also a devcontainer-cli project removing the need for vscode but had all kinds of issues with it so for now i revert to start devcontainer with vscode to debug and test strats and get an issolated solid dev container to debug code and test strats in.

References: https://code.visualstudio.com/docs/devcontainers/containers https://code.visualstudio.com/docs/devcontainers/tutorial

[just-nilux]

Oh nice! Thanks! You should make a Pull Request here in this repo then @nateemma can add it to the repo

[Jarrodsz]

There is a devcontainer in the freqtrade i shall rebase of that

[Jarrodsz]

Oh nice! Thanks! You should make a Pull Request here in this repo then @nateemma can add it to the repo

I think its a little bit complicated to make a pull request for this, im not sure how Phil managed to get it working on an M1 ARM native ( I assume ) I have tried for hours. For the dev container to be used there needs to be several different changes to the repo not sure that is something that is wanted.

But a devcontainer *could help everyone have the same development environment and an easy starting point to develop the strategies or strategies in general.

For example you would probably have to create a user_data/strategies/ folder and map the user_data folder inside the container. Perhaps its more interesting to create a requirements-strats.txt file

So I don't know what the ideal workflow is for developing strategies for me on a mac its certainly the devcontainer road since that seems the only way to go at this moment.

Perhaps we can discuss for now an implementation here ( i will update it in next hour ): https://github.com/Jarrodsz/freqtrade-devcontainer

And if there is interest it can be integrated ( or not ) i assume its a bit outside the scope of this repro but nevertheless could be very usefull since all the dependencies the strats run on.

[nateemma]

In terms of a native Mac-M1 environment, have you looked at user_data/strategies/scripts/install_packages.sh ?

Thanks,

Phil

On Tue, May 2, 2023 at 11:53 PM Jarrodsz @.***> wrote:

Oh nice! Thanks! You should make a Pull Request here in this repo then @nateemma https://github.com/nateemma can add it to the repo

I think its a little bit complicated to make a pull request for this, im not sure how Phil managed to get it working on an M1 ARM native ( I assume ) I have tried for hours. For the dev container to be used there needs to be several different changes to the repo not sure that is something that is wanted.

But a devcontainer *could help everyone have the same development environment and an easy starting point to develop the strategies or strategies in general.

For example you would probably have to create a user_data/strategies/ folder and map the user_data folder inside the container. Perhaps its more interesting to create a requirements-strats.txt file

So I don't know what the ideal workflow is for developing strategies for me on a mac its certainly the devcontainer road since that seems the only way to go at this moment.

Perhaps we can discuss for now an implementation here ( i will update it in next hour ): https://github.com/Jarrodsz/freqtrade-devcontainer

And if there is interest it can be integrated ( or not ) i assume its a bit outside the scope of this repro but nevertheless could be very usefull since all the dependencies the strats run on.

— Reply to this email directly, view it on GitHub https://github.com/nateemma/strategies/issues/22#issuecomment-1532531591, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABD4X5Z5WRLTGYCELKBSRZTXEH6IBANCNFSM6AAAAAAXQP7ASU . You are receiving this because you were mentioned.Message ID: @.***>

[Jarrodsz]

Hi Phil,

Yes i have but installing freqtrade native on m1 is nearly Impossible so those commands would run but i found compiling or installing freqtrade dependencies near impossible, i tried 3 times even as far as compiling most of the librarys from source. Dont have the details its a while back. If you know of a way to install on osx native ( i assume you did reading your comments ) that would be a gamechanger. Docker and devcontainer are a pain to debug code although devcontainer gives me solid installation of all freqtrade dependencies.

Not many people use arm for that reason i guess. Its not popular among python devs. Even installing something like ruby on rails with brew was such a headache a year ago and it still is.

What i did was run the setup script from freqtrade then manually tried to fix all compile dep issues one by one but gave up after 1 hour of breaking deps.

[nateemma]

I almost always have an issue with talib. I get around it by uninstalling and re-installing, like this:

brew uninstall ta-lib brew install ta-lib sh setup.sh -u

The last time I did this, I also had to install sqlite using conda (not pip)

Cheers,

Phil

On Wed, May 3, 2023 at 8:42 AM Jarrodsz @.***> wrote:

Hi Phil,

Yes i have but installing freqtrade native on m1 is nearly Impossible so those commands would run but i found compiling or installing freqtrade dependencies near impossible, i tried 3 times even as far as compiling most of the librarys from source. Dont have the details its a while back. If you know of a way to install on osx native ( i assume you did reading your comments ) that eoild be a gamechanger. Docker and devcontainer are a pain to debig code although devcontainer gives me solid installation of all freqtrade dependencies.

Not many people use arm for that reason i guess. Its not popular among python devs.

What i did was run the setup script from freqtrade then manually tried to fix all compile dep issues one by one but gave up after 1 hour of breaking deps.

— Reply to this email directly, view it on GitHub https://github.com/nateemma/strategies/issues/22#issuecomment-1533270592, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABD4X53LE7GTOMZXKTT4VMLXEJ4E7ANCNFSM6AAAAAAXQP7ASU . You are receiving this because you were mentioned.Message ID: @.***>

[just-nilux]

I prefer to run a vm with ubuntu and use this for my python development. You don't have any issues with dependencies and compatibility, and also you keep your mac libraries clean. Development can amass a ton of unnecessary libraries and you don't need them in your mac.

[Jarrodsz]

Development

Sounds reasonable, did not even think about that yet. Any suggestions on what to use? Im using devcontainers now but still not really happy with it. You use parallels instead? Using a vm like parallels in fullscreen sounds actually like a really nice solution to have a native feel and no longer need to bother with docker ( only to run in production) i think there is even an intellij ultimate ide for linux so rhat would be awesome.