Closed swilson314 closed 4 years ago
I fixed the typo in the first case. It is interesting it was passing all the builds and tests
Re/ other exceptions, I cannot see a problem immediately. None of them has an assert call? What kind of exception do they throw?
Is this a C++17 enabled build? Do you get any warnings?
Yes, the /std:c++17 is set.
Here are the warnings I get:
1>------ Build started: Project: dataframe_tester, Configuration: Debug Win32 ------
1>dataframetester.cc
1>f:\dev\dataframe-master\include\dataframe\vectors\vectorptrview.h(59): warning C4251: 'hmdf::VectorPtrView
Here is the exception/call stack for the first error:
Second error:
It is hard to debug like this. A couple of lines in your stack trace corresponds to writing to std::cout which is strange. Also, this passes the Appveyor which I think is Windows 10. And in Appveyor, I see all the warnings you listed.
I think you are also turning on a switch that triggers STL debug code?
As an experiment to eliminate one possible cause:
Can you comment out all the calls to MyDataFrame::set_thread_level()
and the corresponding asserts with MyDataFrame::get_thread_level()
?
Here are the compiler flags, I haven't changed anything from the original cmake creation of the project:
/GS /TP /analyze- /W3 /Zc:wchar_t /I"F:\Dev\DataFrame-master\include" /Zi /Gm- /Od /Ob0 /Fd"F:\Dev\DataFrame-master\debug\bin\Debug\dataframe_tester.pdb" /Zc:inline /fp:precise /D "WIN32" /D "_WINDOWS" /D "_CRT_SECURE_NO_WARNINGS" /D "LIBRARY_EXPORTS" /D "CMAKE_INTDIR=\"Debug\"" /D "_MBCS" /errorReport:prompt /WX- /Zc:forScope /RTC1 /GR /Gd /Oy- /MDd /std:c++17 /FC /Fa"Debug/" /EHsc /nologo /Fo"dataframe_tester.dir\Debug\" /Fp"dataframe_tester.dir\Debug\dataframe_tester.pch" /diagnostics:classic
Can you also run it with release mode
cmake -DCMAKE_BUILD_TYPE=Release ..
aside from the elimination of MyDataFrame::get_thread_level()
calls
No change with removal of set_thread_level().
Works fine in Release build. This includes the clearly bad stmt:
assert(dvec[5] == 2.2345)
because NDEBUG is defined, so assert() is nop.
hmm, all these works fine in Linux which assert is really assert. This might to do that that dll's are not properly built for this project.
You may want to use the Conan package which is up to release 1.8.0 CC @yssource
If you're not getting an error in a linux debug build with the stmt assert(dvec[5] == 2.2345), then it seems likely your stl isn't doing bounds checking. Maybe this is something you can turn on?
Aside: assert() can be turned on by not specifying NDEBUG in your cmake.
The first error in test_shifting_up_down() occurs in this stmt "if (iter + n < vec_end)" because the result of iter+n is beyond the end of the vector, which I believe is undefined per the standard. While the code works if not strictly checked, it might be better to break this into two loops anyways to avoid "if" testing within the loop.
template<typename I, typename H>
template
using value_type =
typename std::remove_reference<decltype(vec)>::type::value_type;
const auto vec_end = vec.end();
for (auto iter = vec.begin(); iter != vec_end; ++iter)
if (iter + n < vec_end)
*iter = std::move(*(iter + n));
else
*iter = std::move(_get_nan<value_type>());
return;
}
https://github.com/hosseinmoein/DataFrame/issues/64
if(MSVC)
add_definitions(-DLIBRARY_EXPORTS)
endif()
@hosseinmoein By the way, Conan does not do any works on unit testing. It has a patch removing all the testing bins on the CMakeLists.txt. So, Conan does not verify dataframe_tester and other test bins.
@swilson314 , I fixed the iterator arithmetic to be standard. Please try it again and let me know
@hosseinmoein By the way, Conan does not do any works on unit testing. It has a patch removing all the testing bins on the CMakeLists.txt. So, Conan does not verify dataframe_tester and other test bins.
Understood
Certainly got further this time :)
First exception is now in test_fill_missing_fill_linear_interpolation():
_NODISCARD _Ty& operator[](const size_type _Pos)
{ // subscript mutable sequence
_STL_VERIFY(_Pos < size(), "vector subscript out of range");
return (this->_Myfirst()[_Pos]);
}
From code, call is vec[14], but size of vec is 14:
This is also fixed. I have to get one of those compilers :-)
If you have a windows VM, there’s a “community” version of visual studio.
Also, I did run through the remaining code and there were more exceptions thrown. I stopped after about 5.
Could you send me the location of other exceptions?
Sure thing, will work on this tomorrow.
test_gen_datetime_index()
test_remove_data_by_sel() _STL_VERIFY(_Off <= _Mycont->_Mylast - _Ptr, "cannot seek vector iterator after end");
test_drop_missing_all_2_drop()
_NODISCARD reference operator*() const
{ // return designated value
const auto _Mycont = static_cast<const _Mytree *>(this->_Getcont());
_STL_ASSERT(_Mycont, "cannot dereference value-initialized map/set iterator");
_STL_VERIFY(this->_Ptr != _Mycont->_Myhead, "cannot dereference end map/set iterator");
PS Not sure why the last one failed "out of order", i.e. not the first time it ran.
test_get_data_by_rand()
Here's another one that failed "out of order" ... maybe race with threads?
test_drop_missing_all_no_drop()
_NODISCARD reference operator*() const
{ // return designated value
const auto _Mycont = static_cast<const _Mytree *>(this->_Getcont());
_STL_ASSERT(_Mycont, "cannot dereference value-initialized map/set iterator");
_STL_VERIFY(this->_Ptr != _Mycont->_Myhead, "cannot dereference end map/set iterator");
test_get_view_by_rand()
test_drop_missing_any()
_NODISCARD reference operator*() const
{ // return designated value
const auto _Mycont = static_cast<const _Mytree *>(this->_Getcont());
_STL_ASSERT(_Mycont, "cannot dereference value-initialized map/set iterator");
_STL_VERIFY(this->_Ptr != _Mycont->_Myhead, "cannot dereference end map/set iterator");
test_view_visitors()
test_k_means()
test_affinity_propagation()
_NODISCARD reference operator*() const
{ // return designated object
const auto _Mycont = static_cast<const _Myvec *>(this->_Getcont());
_STL_VERIFY(_Ptr, "can't dereference value-initialized vector iterator");
_STL_VERIFY(_Mycont->_Myfirst <= _Ptr && _Ptr < _Mycont->_Mylast,
"can't dereference out of range vector iterator");
test_drop_threashold_3()
_NODISCARD reference operator*() const
{ // return designated value
const auto _Mycont = static_cast<const _Mytree *>(this->_Getcont());
_STL_ASSERT(_Mycont, "cannot dereference value-initialized map/set iterator");
_STL_VERIFY(this->_Ptr != _Mycont->_Myhead, "cannot dereference end map/set iterator");
*** The End ***
@swilson314 , thanks a lot Actually today I got an ubuntu laptop with a gcc that is even more strict in debug mode than VC++. I fixed all the asserts in dataframe tester and tester_2. But Two categories still remain that I cannot fix but they are safe. I inspected all of them.
I tried the new version and am still getting exceptions, but I don't know whether they fall in the categories above. Which are the functions that will still fail? Should they be #ifdef'd for the compiler that works?
On your side the first exception should be get_data_by_rand
or k-means
which both fall under the first category.
Any testing of views will throw an exception because of the second category.
The first category is obviously not a problem. The second category is not a problem at run time. It is just violating the STL boundary checks enabled by the compiler flag. You could disable that flag and still have a debug-able executable.
I may have a solution for the second case. But I have to find time to implement it.
Would it make sense to document this in the code? For anybody looking to adopt the library, a first question will be whether the tests/examples work.
Test examples work for every environment that people have reported to me, including many windows environment (of course excluding random number stuff). Is the boundary check flag a default on VC++ 2017?
I should document the random number algos
@swilson314 , My solution worked to fix the views violating STL boundary checks. I also removed asserts from testing algorithms based on random numbers. So, you now should be able run all tests clean through
Fantastic!! I’ll give it a try. I’m working on an example that loads a DataFrame class from an Arrow Table, read from a csv file. Should be able to post next week.
I still get an assert here:
Are you located in EST region? I also suspect timezone info might not be updated on your host.
Also: test_drop_missing_all_2_drop() test_drop_missing_all_no_drop() test_view_visitors() test_drop_missing_any() test_drop_threashold_3()
I'm PST
I've built DataFrame with VS 2017 on Windows 10. A number of exceptions are thrown when I run dataframe_tester.exe.
The first occurs here, I think this test is wrong since the size of dvec is 1:
Other exceptions occur during the following tests, I stopped testing at this point: