Please read the submitted article and fully complete this form. Since we don't
have a copy editor, we also request that you annotate the PDF [1]_ to highlight
typos, formatting issues, and grammatical mistakes.
The goal of the review process is two-fold. First, it guides authors in
improving their papers and, secondly, ensures that published works are of a
professional academic standard.
Research in science and engineering increasingly relies on software for
data processing and management as well as theoretical exploration. However,
the effort necessary to develop this software is rarely recognized as having
the same academic worth as other aspects of the research. These proceedings
are, at least in part, intended to address this shortcoming.
An article focused on software development necessarily differs from the
standard scientific article with respect to format. For instance, it is
unlikely to have the same sections (i.e., introduction, methods, results,
conclusion). You may therefore have to rely on other factors to decide whether
the paper sets a high enough standards as an academic publication.
Please note that, while reviewers' recommendations regarding a paper's
suitability for publication are seriously considered, the final decision rests
with the proceeding editors.
.. [1] We recommend the free version of PDF XChange Viewer <http://www.tracker-software.com/product/pdf-xchange-viewer> for
Linux (Wine) and Windows. Under OSX, annotation is provided by Preview
as well as Skim <http://skim-app.sourceforge.net/>.
GENERAL EVALUATION
Please rate the paper using the following criteria (please use the abbreviation
to the right of the description)::
below doesn't meet standards for academic publication
meets meets or exceeds the standards for academic publication
n/a not applicable
Quality of the approach: below
Quality of the writing: below
Quality of the figures/tables: below
SPECIFIC EVALUATION
For the following questions, please respond with 'yes' or 'no'. If you
answer 'no', please provide a brief, one- to two-sentence explanation.
Is the code made publicly available and does the article sufficiently
describe how to access it? We aim not to publish papers that essentially
advertise proprietary software. Therefore, if the code is not publicly
available, please provide a one- to two- sentence response to each of the
following questions:
Does the article focus on a topic other than the features
of the software itself?
Can the majority of statements made be externally validated
(i.e., without the use of the software)?
Is the information presented of interest to readers other than
those at whom the software is aimed?
Is there any other aspect of the article that would
justify including it despite the fact that the code
isn't available?
Does the article discuss the reasons the software is closed?
Yes, the code is publicly available and relies on a variety of open source
packages. The software can be downloaded from the github link provided in the
paper. The Android image capture library may not be open source.
Does the article present the problem in an appropriate context?
Yes.
Specifically, does it:
explain why the problem is important,
Yes, he gives background on self-driving car research to show the need.
describe in which situations it arises,
outline relevant previous work,
Maybe, only a couple of papers are cited about self-driving vehicles and
techniques. Much of the literature is not present and no detailed commentary on
how the method presented in the paper compares to other approaches.
provide background information for non-experts
Yes, the paper seems to be written with a non-expert audience in mind.
Is the content of the paper accessible to a computational scientist
with no specific knowledge in the given field?
Yes.
Does the paper describe a well-formulated scientific or technical
achievement?
No, the paper doesn't propose a hypothesis or follow the scientific method. It
is more like a report on how to use several software libraries to accomplish a
task rather than a well-formulated scientific achievement. I guess it can be
called a "technical achievement" because the author achieved his goal of
reproducing another's work with different software and hardware.
Are the technical and scientific decisions well-motivated and
clearly explained?
No, the motivation only seems to be to replicate previous work. The reasons for
choosing the hardware, software, and parameters for both are not explained at
all. It seems as if the author just used informed guesses at values for the
neural network, for example.
Are the code examples (if any) sound, clear, and well-written?
Yes, but it'd be nice if they followed PEP8 standards for readability reasons.
Is the paper factual correct?
Yes, the method and results seem to be factually correct.
Is the language and grammar of sufficient quality?
Yes.
Are the conclusions justified?
No, because there are no conclusions. The paper simply describes how something
is done. Most scientific works explain what the conclusions are and make some
reasoning on why those conclusions are true. The conclusion here seems to be
simply that someone else's work can be replicated with different hardware and
software while using the same methods.
Is prior work properly and fully cited?
No. The blog post that described the work that is being replicated is cited
with a URL and a few academic papers are cited, but the wide berth of work on
self-driving vehicles and neural networks has been ignored.
Should any part of the article be shortened or expanded? Please explain.
Yes. The prior literature on the subject should be expanded and comparisons in
this method and others is needed. Furthermore, there should be some scientific
discourse on the details of this method along with quantitative measures
describing the performance of this technique so that comparisons can be made to
other software, hardware, and methods.
In your view, is the paper fit for publication in the conference proceedings?
Please suggest specific improvements and indicate whether you think the
article needs a significant rewrite (rather than a minor revision).
This paper presents the replication of a "self-driving" robot vehicle
implementation that utilizes a trained neural network to follow a specific
course using visual inputs to control the vehicle's driving motors. From what
is written, it seems that using a Python based software stack and the Lego
robot kit, that prior work can be replicated. But the article does not exhibit
the depth that other quality scientific articles on this subject offer. The
reader is simply instructed in the how, i.e. the method, of implementing this
system using a very specific selection of software packages. Little to no
information is provided that gives the reader technical information on the
capabilities of this method, particularly not for comparison purposes to other
methods. No hypothesis or claim is made nor any proof to back it up the missing
claim. The article seems more akin to a undergraduate lab tutorial that simply
shows the student how to do something, but misses the "why" portion that
generally makes a contribution interesting and publishable for the scientific
community. I think this article could be transformed into a valuable scientific
contribution if these things were changed/added:
A proper literature review on other methods. At the minimum, this could
detail other software libraries with these capabilities and at the maximum
this could include comparison to other methods of autonomously controlling a
vehicle.
A statement, hypothesis, or claim about what makes this method
special/difference and the proof to back it up. If this is simply a
replication study of previous work using different methods, then the claims
from the previous study should also be proved by the method presented in
this paper along with detailed quantitative comparisons of how well the
other method was replicated.
More technical detail on the method. If comparing software, we need to know
things like how easy it is to use, how fast it is, how robust it is, what
are the limitations, etc all with respect to other available software. The
technical details of the hardware are also important so that we no its
advantages and limitations with respect to other methods. If comparing
algorithms (neural nets, etc) then we need to know more detail about the
methods and why the parameters you chose are good and what they mean.
Explanation of the neural net framework you chose and why would be helpful.
Give accurate and precise results. Simply saying that your vehicle completes
the course "about 2/3rds" of the time is not science. You also need
dimensional descriptions of the course, the vehicle, and metrics on how bad
or good it actually performs. No one can compare their vehicle and
implementation to yours if this isn't provided. We also have no idea if you
actually replicated the prior work because there are no quantitative
measures.
The tone and grammar of the paper resembles a blog post as opposed to a
scientific article. I'm not opposed to having more personal writing, but it
needs to be justified and it needs to contribute to the understanding of the
article. As it stands, this would be a fine blog post but it is quite far
from an average scientific article.
Independent Review Report
.. note:: Please be aware that all reviews are made public including the reviewer's name.
Reviewer: Jason K. Moore
Department/Center/Division: Human Motion and Control Lab, Mechanical Engineering Department
Institution/University/Company: Cleveland State University
Field of interest / expertise: multibody dynamics, biomechanics, control systems, system identification
Country: USA
Article reviewed: Self-driving Lego Mindstorms Robot
INSTRUCTIONS
Please read the submitted article and fully complete this form. Since we don't have a copy editor, we also request that you annotate the PDF [1]_ to highlight typos, formatting issues, and grammatical mistakes.
The goal of the review process is two-fold. First, it guides authors in improving their papers and, secondly, ensures that published works are of a professional academic standard.
Research in science and engineering increasingly relies on software for data processing and management as well as theoretical exploration. However, the effort necessary to develop this software is rarely recognized as having the same academic worth as other aspects of the research. These proceedings are, at least in part, intended to address this shortcoming.
An article focused on software development necessarily differs from the standard scientific article with respect to format. For instance, it is unlikely to have the same sections (i.e., introduction, methods, results, conclusion). You may therefore have to rely on other factors to decide whether the paper sets a high enough standards as an academic publication.
Please note that, while reviewers' recommendations regarding a paper's suitability for publication are seriously considered, the final decision rests with the proceeding editors.
.. [1] We recommend the free version of
PDF XChange Viewer <http://www.tracker-software.com/product/pdf-xchange-viewer>for Linux (Wine) and Windows. Under OSX, annotation is provided by Preview as well asSkim <http://skim-app.sourceforge.net/>.GENERAL EVALUATION
Please rate the paper using the following criteria (please use the abbreviation to the right of the description)::
below doesn't meet standards for academic publication meets meets or exceeds the standards for academic publication n/a not applicable
SPECIFIC EVALUATION
For the following questions, please respond with 'yes' or 'no'. If you answer 'no', please provide a brief, one- to two-sentence explanation.
Yes, the code is publicly available and relies on a variety of open source packages. The software can be downloaded from the github link provided in the paper. The Android image capture library may not be open source.
Yes.
Specifically, does it:
Yes, he gives background on self-driving car research to show the need.
Maybe, only a couple of papers are cited about self-driving vehicles and techniques. Much of the literature is not present and no detailed commentary on how the method presented in the paper compares to other approaches.
Yes, the paper seems to be written with a non-expert audience in mind.
Yes.
No, the paper doesn't propose a hypothesis or follow the scientific method. It is more like a report on how to use several software libraries to accomplish a task rather than a well-formulated scientific achievement. I guess it can be called a "technical achievement" because the author achieved his goal of reproducing another's work with different software and hardware.
No, the motivation only seems to be to replicate previous work. The reasons for choosing the hardware, software, and parameters for both are not explained at all. It seems as if the author just used informed guesses at values for the neural network, for example.
Yes, but it'd be nice if they followed PEP8 standards for readability reasons.
Yes, the method and results seem to be factually correct.
Yes.
No, because there are no conclusions. The paper simply describes how something is done. Most scientific works explain what the conclusions are and make some reasoning on why those conclusions are true. The conclusion here seems to be simply that someone else's work can be replicated with different hardware and software while using the same methods.
No. The blog post that described the work that is being replicated is cited with a URL and a few academic papers are cited, but the wide berth of work on self-driving vehicles and neural networks has been ignored.
Yes. The prior literature on the subject should be expanded and comparisons in this method and others is needed. Furthermore, there should be some scientific discourse on the details of this method along with quantitative measures describing the performance of this technique so that comparisons can be made to other software, hardware, and methods.
This paper presents the replication of a "self-driving" robot vehicle implementation that utilizes a trained neural network to follow a specific course using visual inputs to control the vehicle's driving motors. From what is written, it seems that using a Python based software stack and the Lego robot kit, that prior work can be replicated. But the article does not exhibit the depth that other quality scientific articles on this subject offer. The reader is simply instructed in the how, i.e. the method, of implementing this system using a very specific selection of software packages. Little to no information is provided that gives the reader technical information on the capabilities of this method, particularly not for comparison purposes to other methods. No hypothesis or claim is made nor any proof to back it up the missing claim. The article seems more akin to a undergraduate lab tutorial that simply shows the student how to do something, but misses the "why" portion that generally makes a contribution interesting and publishable for the scientific community. I think this article could be transformed into a valuable scientific contribution if these things were changed/added: