Closed raybooysen closed 4 years ago
odeheurles
commented
8 years ago I think what is really slow is updating the dependencies npm i and dnu restore and also downloading base docker images used by the build.
@tdeheurles do you know where time is spent and how we could improve this?
tdeheurles
commented
8 years ago The circle ci tests can be run locally. So the build time can be reduced to the concerned service. Cd deploy/docker && testAll.
There is around 1min/1min30 of circleci time: loading/saving cache.
Npm cache seems good only when node_module is already present (building with .npm saved is still 3 minutes).
Dnx is around 10/15 seconds with dnx cache now.
I give better time feedback when home.
But for me, a part of the time is to have a good feedback of project creation too. When circleci goes green, I also have a good idea of building locally with the container way.
Will have a look to this in the coming days.
tdeheurles
commented
8 years ago - Start the build: 00:45
- Restore cache: 01:00
- Build: 03:00
- Tests: 00:15
- Save cache: 00:40
- Stop: 00:10
MASTER:
- Pushing : 07:00I don't have the Build details for now but the time order is 2:30 for node.
raybooysen
commented
8 years ago What is master 7mins?
Also you say 2:30 for node but that isn't in the breakdown.
From: thibault deheurles
Sent: Sunday 27 December 19:46
Subject: Re: [ReactiveTraderCloud] Reduce Build times (#126)
To: AdaptiveConsulting/ReactiveTraderCloud
Cc: Ray Booysen
I don't have the Build details for now but the time order is 2:30 for node.
—
Reply to this email directly or view it on GitHub.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP.
tdeheurles
commented
8 years ago It's the push to docker hub. I totally agree that it's really long.
A solution (without changing docker repository) could be to choose which build we want to push. For example something like a deploy branch that will build and push. When we want the artifact, we just merge in that one.
To accelerate all this things, we can:
raybooysen
commented
8 years ago We want two builds. One for the day to day development which is just a build and test. The other that deploys.
On Sun, Dec 27, 2015 at 11:56 AM -0800, "thibault deheurles" notifications@github.com wrote:
It's the push to docker hub.
I totally agree that it's really long.
A solution (without changing docker repository) could be to choose which build we want to push. For example something like a deploy branch that will build and push. When we want the artifact, we just merge in that one.
To accelerate all this things, we can:
have a registry in our cluster (I don't think to the google one but to a manual one) deploy jenkins into our cluster (It should be simple as there are no more dependency with docker build)
— Reply to this email directly or view it on GitHub.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP.
tdeheurles
commented
8 years ago What should I test ? All the tests are on the cluster ...
tdeheurles
commented
8 years ago Just remind that for now, node is not build inside a container. The client build will be the same if I remove docker.
raybooysen
commented
8 years ago What I mean is there should be two builds. The first build does a build of the code, runs any unit tests and exits. That's all.
A separate build should exist just for deployment. All it does is build a production release of the code and deploy it somewhere. On 27 Dec 2015 8:00 p.m., "thibault deheurles" notifications@github.com wrote:
What should I test ? All the tests are on the cluster ...
— Reply to this email directly or view it on GitHub https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167438761 .
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP.
odeheurles
commented
8 years ago That's not how we do it generally, is it?
Usually the CI build compiles, run tests and store build artifacts. A second build can then be triggered manually to deploy.
Here the artifacts are docker images, not a small zip file stored on a teamcity server and you can run the deployment scripts which will schedule a deployement of the docker images to kubernetes. This is the same flow we are usually using, the only reason why it is slow is that we are storing the build artifacts on dockerhub, and we had not anticipated it would be - that - slow. If we co-locate the build server and the docker registry, it should be perfectly fine.
Otherwise we can go with the strategy you propose until we have time to setup a co-located docker registry / build server / kubernetes. But I'm not sure how you do manual build on circleCI.
Olivier
On 27 Dec 2015, at 20:39, Ray Booysen notifications@github.com wrote:
What I mean is there should be two builds. The first build does a build of the code, runs any unit tests and exits. That's all.
A separate build should exist just for deployment. All it does is build a production release of the code and deploy it somewhere. On 27 Dec 2015 8:00 p.m., "thibault deheurles" notifications@github.com wrote:
What should I test ? All the tests are on the cluster ...
— Reply to this email directly or view it on GitHub https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167438761 .
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP. — Reply to this email directly or view it on GitHub.
tdeheurles
commented
8 years ago There is no mono on circleci. I ll make a try but I suppose that the setup will be long too. I will have a look.
CircleCi start a fresh container each time and all the setup is done each time with some cache that accelerate the process. Node is long to setup and mono will be too.
I am not sure, but I really think that docker is not the issue here.
Le dim. 27 déc. 2015 22:19, Olivier Deheurles notifications@github.com a écrit :
That's not how we do it generally, is it?
Usually the CI build compiles, run tests and store build artifacts. A second build can then be triggered manually to deploy.
Here the artifacts are docker images, not a small zip file stored on a teamcity server and you can run the deployment scripts which will schedule a deployement of the docker images to kubernetes. This is the same flow we are usually using, the only reason why it is slow is that we are storing the build artifacts on dockerhub, and we had not anticipated it would be - that - slow. If we co-locate the build server and the docker registry, it should be perfectly fine.
Otherwise we can go with the strategy you propose until we have time to setup a co-located docker registry / build server / kubernetes. But I'm not sure how you do manual build on circleCI.
Olivier
On 27 Dec 2015, at 20:39, Ray Booysen notifications@github.com wrote:
What I mean is there should be two builds. The first build does a build of the code, runs any unit tests and exits. That's all.
A separate build should exist just for deployment. All it does is build a production release of the code and deploy it somewhere. On 27 Dec 2015 8:00 p.m., "thibault deheurles" <notifications@github.com
wrote:
What should I test ? All the tests are on the cluster ...
— Reply to this email directly or view it on GitHub < https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167438761
.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP. — Reply to this email directly or view it on GitHub.
— Reply to this email directly or view it on GitHub https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167442248 .
raybooysen
commented
8 years ago OK understood.
But surely the dev build is not doing release versions of the code. So the artifacts being generated aren't the ones you want anyway? On 27 Dec 2015 9:19 p.m., "Olivier Deheurles" notifications@github.com wrote:
That's not how we do it generally, is it?
Usually the CI build compiles, run tests and store build artifacts. A second build can then be triggered manually to deploy.
Here the artifacts are docker images, not a small zip file stored on a teamcity server and you can run the deployment scripts which will schedule a deployement of the docker images to kubernetes. This is the same flow we are usually using, the only reason why it is slow is that we are storing the build artifacts on dockerhub, and we had not anticipated it would be - that - slow. If we co-locate the build server and the docker registry, it should be perfectly fine.
Otherwise we can go with the strategy you propose until we have time to setup a co-located docker registry / build server / kubernetes. But I'm not sure how you do manual build on circleCI.
Olivier
On 27 Dec 2015, at 20:39, Ray Booysen notifications@github.com wrote:
What I mean is there should be two builds. The first build does a build of the code, runs any unit tests and exits. That's all.
A separate build should exist just for deployment. All it does is build a production release of the code and deploy it somewhere. On 27 Dec 2015 8:00 p.m., "thibault deheurles" <notifications@github.com
wrote:
What should I test ? All the tests are on the cluster ...
— Reply to this email directly or view it on GitHub < https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167438761
.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP. — Reply to this email directly or view it on GitHub.
— Reply to this email directly or view it on GitHub https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167442248 .
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP.
raybooysen
commented
8 years ago Sounds like the entire build setup isn't optimal. Our daily dev builds don't need a container at all. Check out code, install dependencies, compile and then run tests. Nothing more.
On Sun, Dec 27, 2015 at 1:26 PM -0800, "thibault deheurles" notifications@github.com wrote:
There is no mono on circleci. I ll make a try but I suppose that the setup
will be long too. I will have a look.
CircleCi start a fresh container each time and all the setup is done each
time with some cache that accelerate the process. Node is long to setup and
mono will be too.
I am not sure, but I really think that docker is not the issue here.
Le dim. 27 déc. 2015 22:19, Olivier Deheurles notifications@github.com a
écrit :
That's not how we do it generally, is it?
Usually the CI build compiles, run tests and store build artifacts. A
second build can then be triggered manually to deploy.
Here the artifacts are docker images, not a small zip file stored on a
teamcity server and you can run the deployment scripts which will schedule
a deployement of the docker images to kubernetes. This is the same flow we
are usually using, the only reason why it is slow is that we are storing
the build artifacts on dockerhub, and we had not anticipated it would be -
that - slow. If we co-locate the build server and the docker registry, it
should be perfectly fine.
Otherwise we can go with the strategy you propose until we have time to
setup a co-located docker registry / build server / kubernetes. But I'm not
sure how you do manual build on circleCI.
Olivier
On 27 Dec 2015, at 20:39, Ray Booysen notifications@github.com wrote:
What I mean is there should be two builds. The first build does a build
of
the code, runs any unit tests and exits. That's all.
A separate build should exist just for deployment. All it does is build a
production release of the code and deploy it somewhere.
On 27 Dec 2015 8:00 p.m., "thibault deheurles" <notifications@github.com
wrote:
What should I test ? All the tests are on the cluster ...
—
Reply to this email directly or view it on GitHub
<
https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167438761
.
*This email and any attachments to it may be confidential and are
intended
solely for the use of the individual to whom it is addressed. Any views
or
opinions expressed are solely those of the author and do not necessarily
represent those of Adaptive Financial Consulting Ltd. *
*If you are not the intended recipient of this email, you must neither
take
any action based upon its contents, not copy nor show it to anyone.*
*Please contact the sender if you believe you have received this email
in
error.*
*Adaptive Financial Consulting Ltd is a company registered in England
and
Wales. Registered number: 08105955. Registered office: 2nd floor, 11
Leadenhall Street, London EC3 V1LP.*
—
Reply to this email directly or view it on GitHub.
—
Reply to this email directly or view it on GitHub
https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167442248
.
— Reply to this email directly or view it on GitHub.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP.
tdeheurles
commented
8 years ago I suppose a build around 4/5 minutes.
The issue is CircleCi not docker. Docker containers comes with the dependencies that you will have to install with your dev build.
Node is the long part and is not using docker for now.
We don't have tests for that build as the existing one need to be run on the cluster.
If you want to run without container. How do I start eventstore ? How do I start crossbar ?
.....
I think we should just forget and learn the good/bad parts of circleci.
.....
For me the only solution is to have persistence on the CI server. If I have this persistence, the build will be faster.
.....
Another point for day to day is why not using docker locally ? Container does not matter where they run. It s one of the docker strength. We have setup everything and we just have to press the button to run everything... The environment are in the containers.
I think that the ci in this project is nothing more than something that color icon on github ... (Because we are using docker)
Le dim. 27 déc. 2015 22:52, Ray Booysen notifications@github.com a écrit :
OK understood.
But surely the dev build is not doing release versions of the code. So the artifacts being generated aren't the ones you want anyway? On 27 Dec 2015 9:19 p.m., "Olivier Deheurles" notifications@github.com wrote:
That's not how we do it generally, is it?
Usually the CI build compiles, run tests and store build artifacts. A second build can then be triggered manually to deploy.
Here the artifacts are docker images, not a small zip file stored on a teamcity server and you can run the deployment scripts which will schedule a deployement of the docker images to kubernetes. This is the same flow we are usually using, the only reason why it is slow is that we are storing
the build artifacts on dockerhub, and we had not anticipated it would be
that - slow. If we co-locate the build server and the docker registry, it should be perfectly fine.
Otherwise we can go with the strategy you propose until we have time to setup a co-located docker registry / build server / kubernetes. But I'm not sure how you do manual build on circleCI.
Olivier
On 27 Dec 2015, at 20:39, Ray Booysen notifications@github.com wrote:
What I mean is there should be two builds. The first build does a build of the code, runs any unit tests and exits. That's all.
A separate build should exist just for deployment. All it does is build a production release of the code and deploy it somewhere. On 27 Dec 2015 8:00 p.m., "thibault deheurles" < notifications@github.com
wrote:
What should I test ? All the tests are on the cluster ...
— Reply to this email directly or view it on GitHub <
https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167438761
.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP. — Reply to this email directly or view it on GitHub.
— Reply to this email directly or view it on GitHub < https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167442248
.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP.
—
Reply to this email directly or view it on GitHub https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167443765 .
raybooysen
commented
8 years ago Why do we need to start eventstore or crossbar?
tdeheurles
commented
8 years ago Any of those ? Containers ? Services ?
Le dim. 27 déc. 2015 23:15, Ray Booysen notifications@github.com a écrit :
Why do you need to start any of those?
On Sun, Dec 27, 2015 at 1:26 PM -0800, "thibault deheurles" < notifications@github.com> wrote:
There is no mono on circleci. I ll make a try but I suppose that the setup
will be long too. I will have a look.
CircleCi start a fresh container each time and all the setup is done each
time with some cache that accelerate the process. Node is long to setup and
mono will be too.
I am not sure, but I really think that docker is not the issue here.
Le dim. 27 déc. 2015 22:19, Olivier Deheurles notifications@github.com a
écrit :
That's not how we do it generally, is it?
Usually the CI build compiles, run tests and store build artifacts. A
second build can then be triggered manually to deploy.
Here the artifacts are docker images, not a small zip file stored on a
teamcity server and you can run the deployment scripts which will schedule
a deployement of the docker images to kubernetes. This is the same flow we
are usually using, the only reason why it is slow is that we are storing
the build artifacts on dockerhub, and we had not anticipated it would be
that - slow. If we co-locate the build server and the docker registry, it
should be perfectly fine.
Otherwise we can go with the strategy you propose until we have time to
setup a co-located docker registry / build server / kubernetes. But I'm not
sure how you do manual build on circleCI.
Olivier
On 27 Dec 2015, at 20:39, Ray Booysen notifications@github.com wrote:
What I mean is there should be two builds. The first build does a build
of
the code, runs any unit tests and exits. That's all.
A separate build should exist just for deployment. All it does is build a
production release of the code and deploy it somewhere.
On 27 Dec 2015 8:00 p.m., "thibault deheurles" < notifications@github.com
wrote:
What should I test ? All the tests are on the cluster ...
—
Reply to this email directly or view it on GitHub
<
https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167438761
.
*This email and any attachments to it may be confidential and are
intended
solely for the use of the individual to whom it is addressed. Any views
or
opinions expressed are solely those of the author and do not necessarily
represent those of Adaptive Financial Consulting Ltd. *
*If you are not the intended recipient of this email, you must neither
take
any action based upon its contents, not copy nor show it to anyone.*
*Please contact the sender if you believe you have received this email
in
error.*
*Adaptive Financial Consulting Ltd is a company registered in England
and
Wales. Registered number: 08105955. Registered office: 2nd floor, 11
Leadenhall Street, London EC3 V1LP.*
—
Reply to this email directly or view it on GitHub.
—
Reply to this email directly or view it on GitHub
< https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167442248
.
— Reply to this email directly or view it on GitHub.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP.
—
Reply to this email directly or view it on GitHub https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167444804 .
tdeheurles
commented
8 years ago Only tests are end to end
Le dim. 27 déc. 2015 23:17, Ray Booysen notifications@github.com a écrit :
Why do we need to start eventstore or crossbar?
— Reply to this email directly or view it on GitHub https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167444843 .
AdaptiveBuild
commented
8 years ago What do you mean by release? A .NET release build and a nodejs build with minification, etc?
I believe CircleCI always run release builds.
ᐧ
On Sun, Dec 27, 2015 at 9:52 PM, Ray Booysen notifications@github.com wrote:
OK understood.
But surely the dev build is not doing release versions of the code. So the artifacts being generated aren't the ones you want anyway? On 27 Dec 2015 9:19 p.m., "Olivier Deheurles" notifications@github.com wrote:
That's not how we do it generally, is it?
Usually the CI build compiles, run tests and store build artifacts. A second build can then be triggered manually to deploy.
Here the artifacts are docker images, not a small zip file stored on a teamcity server and you can run the deployment scripts which will schedule a deployement of the docker images to kubernetes. This is the same flow we are usually using, the only reason why it is slow is that we are storing
the build artifacts on dockerhub, and we had not anticipated it would be
that - slow. If we co-locate the build server and the docker registry, it should be perfectly fine.
Otherwise we can go with the strategy you propose until we have time to setup a co-located docker registry / build server / kubernetes. But I'm not sure how you do manual build on circleCI.
Olivier
On 27 Dec 2015, at 20:39, Ray Booysen notifications@github.com wrote:
What I mean is there should be two builds. The first build does a build of the code, runs any unit tests and exits. That's all.
A separate build should exist just for deployment. All it does is build a production release of the code and deploy it somewhere. On 27 Dec 2015 8:00 p.m., "thibault deheurles" < notifications@github.com
wrote:
What should I test ? All the tests are on the cluster ...
— Reply to this email directly or view it on GitHub <
https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167438761
.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP. — Reply to this email directly or view it on GitHub.
— Reply to this email directly or view it on GitHub < https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167442248
.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP.
— Reply to this email directly or view it on GitHub https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167443765 .
Olivier Deheurles Director +44 (0) 7872 490 812
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP.
raybooysen
commented
8 years ago Yes. But I doubt it knows which NPM script to run to output our prod artifacts through webpack.
From: AdaptiveBuild
Sent: Sunday 27 December 22:37
Subject: Re: [ReactiveTraderCloud] Reduce Build times (#126)
To: AdaptiveConsulting/ReactiveTraderCloud
Cc: Ray Booysen
What do you mean by release? A .NET release build and a nodejs build with
minification, etc?
I believe CircleCI always run release builds.
ᐧ
On Sun, Dec 27, 2015 at 9:52 PM, Ray Booysen notifications@github.com
wrote:
OK understood.
But surely the dev build is not doing release versions of the code. So the
artifacts being generated aren't the ones you want anyway?
On 27 Dec 2015 9:19 p.m., "Olivier Deheurles" notifications@github.com
wrote:
That's not how we do it generally, is it?
Usually the CI build compiles, run tests and store build artifacts. A
second build can then be triggered manually to deploy.
Here the artifacts are docker images, not a small zip file stored on a
teamcity server and you can run the deployment scripts which will
schedule
a deployement of the docker images to kubernetes. This is the same flow
we
are usually using, the only reason why it is slow is that we are storing
the build artifacts on dockerhub, and we had not anticipated it would be
that - slow. If we co-locate the build server and the docker registry, it
should be perfectly fine.
Otherwise we can go with the strategy you propose until we have time to
setup a co-located docker registry / build server / kubernetes. But I'm
not
sure how you do manual build on circleCI.
Olivier
On 27 Dec 2015, at 20:39, Ray Booysen notifications@github.com
wrote:
What I mean is there should be two builds. The first build does a build
of
the code, runs any unit tests and exits. That's all.
A separate build should exist just for deployment. All it does is
build a
production release of the code and deploy it somewhere.
On 27 Dec 2015 8:00 p.m., "thibault deheurles" <
notifications@github.com
wrote:
What should I test ? All the tests are on the cluster ...
—
Reply to this email directly or view it on GitHub
<
https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167438761
.
*This email and any attachments to it may be confidential and are
intended
solely for the use of the individual to whom it is addressed. Any views
or
opinions expressed are solely those of the author and do not
necessarily
represent those of Adaptive Financial Consulting Ltd. *
*If you are not the intended recipient of this email, you must neither
take
any action based upon its contents, not copy nor show it to anyone.*
*Please contact the sender if you believe you have received this email
in
error.*
*Adaptive Financial Consulting Ltd is a company registered in England
and
Wales. Registered number: 08105955. Registered office: 2nd floor, 11
Leadenhall Street, London EC3 V1LP.*
—
Reply to this email directly or view it on GitHub.
—
Reply to this email directly or view it on GitHub
<
https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167442248
.
*This email and any attachments to it may be confidential and are intended
solely for the use of the individual to whom it is addressed. Any views or
opinions expressed are solely those of the author and do not necessarily
represent those of Adaptive Financial Consulting Ltd. *
*If you are not the intended recipient of this email, you must neither
take
any action based upon its contents, not copy nor show it to anyone.*
*Please contact the sender if you believe you have received this email in
error.*
*Adaptive Financial Consulting Ltd is a company registered in England and
Wales. Registered number: 08105955. Registered office: 2nd floor, 11
Leadenhall Street, London EC3 V1LP.*
—
Reply to this email directly or view it on GitHub
https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167443765
.
Olivier Deheurles
Director
+44 (0) 7872 490 812
*This email and any attachments to it may be confidential and are intended
solely for the use of the individual to whom it is addressed. Any views or
opinions expressed are solely those of the author and do not necessarily
represent those of Adaptive Financial Consulting Ltd. *
*If you are not the intended recipient of this email, you must neither take
any action based upon its contents, not copy nor show it to anyone.*
*Please contact the sender if you believe you have received this email in
error.*
*Adaptive Financial Consulting Ltd is a company registered in England and
Wales. Registered number: 08105955. Registered office: 2nd floor, 11
Leadenhall Street, London EC3 V1LP.*
—
Reply to this email directly or view it on GitHub.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP.
AdaptiveBuild
commented
8 years ago Looking at the dev tools on the deployed version I think you're right...
I just had a look in chrome dev tools at app.js. Is that the file produced by webpack?
Do you know what is the webpack command line Thibault should use on the build? ᐧ
On Sun, Dec 27, 2015 at 10:43 PM, Ray Booysen notifications@github.com wrote:
Yes. But I doubt it knows which NPM script to run to output our prod artifacts through webpack.
From: AdaptiveBuild
Sent: Sunday 27 December 22:37
Subject: Re: [ReactiveTraderCloud] Reduce Build times (#126)
To: AdaptiveConsulting/ReactiveTraderCloud
Cc: Ray Booysen
What do you mean by release? A .NET release build and a nodejs build with
minification, etc?
I believe CircleCI always run release builds.
ᐧ
On Sun, Dec 27, 2015 at 9:52 PM, Ray Booysen notifications@github.com
wrote:
OK understood.
But surely the dev build is not doing release versions of the code. So the
artifacts being generated aren't the ones you want anyway?
On 27 Dec 2015 9:19 p.m., "Olivier Deheurles" notifications@github.com
wrote:
That's not how we do it generally, is it?
Usually the CI build compiles, run tests and store build artifacts. A
second build can then be triggered manually to deploy.
Here the artifacts are docker images, not a small zip file stored on a
teamcity server and you can run the deployment scripts which will
schedule
a deployement of the docker images to kubernetes. This is the same flow
we
are usually using, the only reason why it is slow is that we are storing
the build artifacts on dockerhub, and we had not anticipated it would be
that - slow. If we co-locate the build server and the docker registry, it
should be perfectly fine.
Otherwise we can go with the strategy you propose until we have time to
setup a co-located docker registry / build server / kubernetes. But I'm
not
sure how you do manual build on circleCI.
Olivier
On 27 Dec 2015, at 20:39, Ray Booysen notifications@github.com
wrote:
What I mean is there should be two builds. The first build does a build
of
the code, runs any unit tests and exits. That's all.
A separate build should exist just for deployment. All it does is
build a
production release of the code and deploy it somewhere.
On 27 Dec 2015 8:00 p.m., "thibault deheurles" <
notifications@github.com
wrote:
What should I test ? All the tests are on the cluster ...
—
Reply to this email directly or view it on GitHub
<
https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167438761
.
*This email and any attachments to it may be confidential and are
intended
solely for the use of the individual to whom it is addressed. Any views
or
opinions expressed are solely those of the author and do not
necessarily
represent those of Adaptive Financial Consulting Ltd. *
*If you are not the intended recipient of this email, you must neither
take
any action based upon its contents, not copy nor show it to anyone.*
*Please contact the sender if you believe you have received this email
in
error.*
*Adaptive Financial Consulting Ltd is a company registered in England
and
Wales. Registered number: 08105955. Registered office: 2nd floor, 11
Leadenhall Street, London EC3 V1LP.*
—
Reply to this email directly or view it on GitHub.
—
Reply to this email directly or view it on GitHub
<
https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167442248
.
*This email and any attachments to it may be confidential and are intended
solely for the use of the individual to whom it is addressed. Any views or
opinions expressed are solely those of the author and do not necessarily
represent those of Adaptive Financial Consulting Ltd. *
*If you are not the intended recipient of this email, you must neither
take
any action based upon its contents, not copy nor show it to anyone.*
*Please contact the sender if you believe you have received this email in
error.*
*Adaptive Financial Consulting Ltd is a company registered in England and
Wales. Registered number: 08105955. Registered office: 2nd floor, 11
Leadenhall Street, London EC3 V1LP.*
—
Reply to this email directly or view it on GitHub
< https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167443765>
.
Olivier Deheurles
Director
+44 (0) 7872 490 812
*This email and any attachments to it may be confidential and are intended
solely for the use of the individual to whom it is addressed. Any views or
opinions expressed are solely those of the author and do not necessarily
represent those of Adaptive Financial Consulting Ltd. *
*If you are not the intended recipient of this email, you must neither take
any action based upon its contents, not copy nor show it to anyone.*
*Please contact the sender if you believe you have received this email in
error.*
*Adaptive Financial Consulting Ltd is a company registered in England and
Wales. Registered number: 08105955. Registered office: 2nd floor, 11
Leadenhall Street, London EC3 V1LP.*
—
Reply to this email directly or view it on GitHub.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP.
— Reply to this email directly or view it on GitHub https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167445698 .
Olivier Deheurles Director +44 (0) 7872 490 812
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP.
tdeheurles
commented
8 years ago npm run compile is the one used.
What do you mean by circleci know which command to run @ray ? It's not running an automatic build anymore.
raybooysen
commented
8 years ago Coming back to this, what do you mean automatic build is not run anymore?
On 28 December 2015 at 07:41, thibault deheurles notifications@github.com wrote:
npm run compile is the one used.
What do you mean by circleci know which command to run @ray https://github.com/ray ? It's not running an automatic build anymore.
— Reply to this email directly or view it on GitHub https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-167505318 .
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP.
tdeheurles
commented
8 years ago You told that: But I doubt it knows which NPM script to run to output our prod artifacts through webpack. What do you mean by it ?
raybooysen
commented
8 years ago The build
On 4 January 2016 at 14:16, thibault deheurles notifications@github.com wrote:
You told that: But I doubt it knows which NPM script to run to output our prod artifacts through webpack. What do you mean by it ?
— Reply to this email directly or view it on GitHub https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-168688283 .
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP.
tdeheurles
commented
8 years ago Look at this build time.
It have taken 20 minutes to build the same things usually done in 6 minutes.
CIrcleCi have some random in the build time. It's usually longer at the end of the day (GB time), I suppose coming from traffic.
raybooysen
commented
8 years ago Seems CircleCI is a big blocker. Time to look at something else?
On 5 January 2016 at 17:15, thibault deheurles notifications@github.com wrote:
Look at this build https://circleci.com/gh/AdaptiveConsulting/ReactiveTraderCloud/648 time.
It have taken 20 minutes to build the same things usually done in 6 minutes.
CIrcleCi have some random in the build time. It's usually longer at the end of the day (GB time), I suppose coming from traffic.
— Reply to this email directly or view it on GitHub https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-169067126 .
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP.
tdeheurles
commented
8 years ago Yes ^^
I have not tested TravisCI but I suppose we will have the same kind of issue. The build is of the same kind. I will look if it can be tested quickly tomorrow.
But as I say before, I think running a jenkins/teamcity in our cluster is the prefered way. It's on my todo list after deploy documentation and some cleaning of the deploy folder.
tdeheurles
commented
8 years ago 
I have also removed the cache for containers (it take less time to just download).
We will see with some try tomorrow. 4 minutes is not perfect but it's clearly better.
tdeheurles
commented
8 years ago The same build on circle is really longer. Removing cache didnt reduce the time.
Also the build fail (before testing I start containers and then I was saving the containers for cache. That time let the containers start ... A sleep of 5 seconds is enough on travis. On circle it's not sufficient ...). It took 8 minutes to fail where travis did the work in 4 minutes (+1 minute to really trigger and start).
Will confirm that tomorrow.
raybooysen
commented
8 years ago Is there no way to get pushed a change when the container is started? Instead of sleeping?
On 5 January 2016 at 21:25, thibault deheurles notifications@github.com wrote:
The same build on circle is really longer. Removing cache didnt reduce the time.
Also the build fail (before testing I start containers and then I was saving the containers for cache. That time let the containers start ... A sleep of 5 seconds is enough on travis. On circle it's not sufficient ...). It took 8 minutes to fail where travis did the work in 4 minutes (+1 minute to really trigger and start).
Will confirm that tomorrow.
— Reply to this email directly or view it on GitHub https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-169138593 .
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP.
tdeheurles
commented
8 years ago It's not the container himself but the c# or nginx inside that need to be started.
I will look for cleaner solution. (I can make the curl test try several time).
Le mar. 5 janv. 2016 22:27, Ray Booysen notifications@github.com a écrit :
Is there no way to get pushed a change when the container is started? Instead of sleeping?
On 5 January 2016 at 21:25, thibault deheurles notifications@github.com wrote:
The same build on circle is really longer. Removing cache didnt reduce the time.
Also the build fail (before testing I start containers and then I was saving the containers for cache. That time let the containers start ... A sleep of 5 seconds is enough on travis. On circle it's not sufficient ...). It took 8 minutes to fail where travis did the work in 4 minutes (+1 minute to really trigger and start).
Will confirm that tomorrow.
— Reply to this email directly or view it on GitHub < https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-169138593
.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Adaptive Financial Consulting Ltd.
If you are not the intended recipient of this email, you must neither take any action based upon its contents, not copy nor show it to anyone.
Please contact the sender if you believe you have received this email in error.
Adaptive Financial Consulting Ltd is a company registered in England and Wales. Registered number: 08105955. Registered office: 2nd floor, 11 Leadenhall Street, London EC3 V1LP.
—
Reply to this email directly or view it on GitHub https://github.com/AdaptiveConsulting/ReactiveTraderCloud/issues/126#issuecomment-169138970 .
tdeheurles
commented
8 years ago Sleep is removed now.
Last test:
As it's free, I propose to have the two for a moment in order to have a better feedback. But I will only push with travis as it's the faster one.
DimitarChristoff
commented
8 years ago travis is awesome, I've always used that. +1
tdeheurles
commented
8 years ago 2 or 3 month ago, I generated a vagrant project where the CI was trying to:
The ssh feature is bugged on travis. It's working on CircleCI. As for now we don't need to connect into these containers (other project was more a DevOps project where I needed to control the build). So I think it's fine with Travis.
I have initially chosen Circle for this project in regards of that ssh issue. Didn't know that circle was comming with a bit of random execution time ...
tdeheurles
commented
8 years ago The build on jenkins have been setup with the hive build system. For now it's still in progress as some bugs need to be fixed, some cleanup to be done and the documentation to be updated. Also, as jenkins run in a container, hive add features to simplify everything and it's to soon to integrate in reactive-trader.
The build can be seen here: https://jenkins.adaptivecluster.com (create a user and ping me to be granted).
A build is:
less than 3 minutes for building and testing everythingbuild + push is around 7 minutes.Note that for now jenkins does not run on our kubernetes cluster. I'm using docker volume from docker version 1.9 while GoogleContainerEngine is still using 1.8. The new volume system is really simpler and I expect Google Container Engine to update soon.
On the other hand, having the jenkins instance segregated from kubernetes is certainly a better solution as for now, the jenkins instance needs a direct access to the node docker client. There are cerainly security issues in this design.
Note that the only things installed on jenkins are git and some plugin to manage jenkins like post-build. It's free of build dependencies
tdeheurles
commented
8 years ago This PR address this problem: https://github.com/AdaptiveConsulting/ReactiveTraderCloud/pull/254
Current build times for the client are > 6 minutes.
Some thoughts: 1) This is very long 2) Do we need the entire container infrastructure just to build, run unit tests? 3) Can we split out another build that is just the day to day dev builds so we get quick feedback? 4) Surely CircleCI allows us to build on pre-ready machines (ones with npm and node already installed)