Closed cayetanobv closed 6 years ago
Hi @cvvergara ,
I compiled without Docker your branch test2 and the bad_alloc raises again. I realized that application never enters in this IF statement https://github.com/cvvergara/osm2pgrouting/blob/test2/src/osm_elements/OSMDocument.cpp#L125
if (m_relations.size() % m_chunk_size == 0) {
wait_child();
std::cout << "Current osm_relations:\t" << m_relations.size();
osm_table_export(m_relations, "osm_relations");
m_relPending = false;
}
Is there maybe something wrong with this behaviour?
Thanks,
@cayetanobv I leave it to you as homework:
if it never enters that part of the code where the data gets exported to the database then tell me how come you get relations in the database on the first place?
if the only change is: that test2 has
} else {
m_rDocument.AddRelation(*last_relation);
I still can't replicate your error. bad::alloc is when you run out of memory
So I might have more memory than you
Hi @cvvergara ,
Sorry for the delay in response... too much work in the last week.
I put some prints in include/database/Export2DB.h
to see whats going on and a try/catch block to avoid finish insert data process (and explore database):
https://github.com/cayetanobv/osm2pgrouting/blob/cayetano/test2/include/database/Export2DB.h#L89
void export_osm (
std::vector<T> &items,
const std::string &table) const {
auto osm_table = m_tables.get_table(table);
std::vector<std::string> values(items.size(), "");
try{
size_t i(0);
for (auto it = items.begin(); it != items.end(); ++it, ++i) {
auto item = *it;
values[i] = tab_separated(item.values(osm_table.columns(), true));
if (table == "osm_relations"){
std::cout << "\nNEW ROW XXXX\t" << values[i];
std::cout << "\n-- XXXX\t" << i;
}
}
} catch (const std::bad_alloc& e) {
std::cout << "\nERROR: " << e.what() << '\n';
}
export_osm(values, osm_table);
}
Shell output is too long with this prints, so I attached this file with all output info: output.txt
Bad_alloc raises after relation with osm_id=73447
is inserted in the database (you can see that this is a very big osm relation).
Thanks!
Ok, Well, because its the fist time it really spits out relations, we will use the one that works for you. Maybe when using libosmium we can fix that kind of big relations issue. But not now.
Ok @cvvergara , but... do we keep this try/catch block inside export_osm()
function (removing innecesary prints) to allow the insertion to database of all possible relations until some error raises or we mantain your commit ?
Work in progress for this issue in: https://github.com/cvvergara/osm2pgrouting/pull/13
We can close this issue @cvvergara . This is fixed with this PR https://github.com/pgRouting/osm2pgrouting/pull/212
Is this problem solved? I converted this problem, is not too much data, osm data 3g, osm2pgrouting version 2.3.3.
Converted data size can not exceed 2g?
print:
Exporting configuration ...
Done Opening data file: usa1_ways.osm total lines: 77327590 Parsing data std::bad_alloc
@cvvergara @cayetanobv
Thanks!
Hi @hetongde , the problem is solved in the context of this issue, but if you want I could test your config with your data. I would need the following:
osm2pgrouting
shell command with all the flags you are using.REMARK: I would say that everything is a matter of RAM memory in this version yet. In future versions this issue will be improved (cc: @cvvergara).
Thanks,
Thanks verymuch @cayetanobv ,i using osmconvert convert us-northeast-latest.osm.pdf to osm,then using osmfilter extract highway from osm ,this osm file about 3.5g,then using osm2pgrouting to get topo. 1 . osm2pgrouting sell command is: osm2pgrouting --f usa1_ways.osm --conf "/usr/share/osm2pgrouting/mapconfig_for_cars.xml" --dbname usalarge --username osm --password 123456
2.RAM total used free shared buffers cached Mem: 16063392 4493504 11569888 332008 316988 2873620
Could you pass me exact filter applied to osmfilter (copy and paste shell command) to get same data as you? Thanks.
Ok, @cayetanobv There are two main steps:
@hetongde I dont think that osm2pgRouting can handle 2G in memory, I think you need to break the file into sections and load it by parts. (unless you are using a very big server) So, break your bounding box into 4, a little overlapping, bounding boxes and load the 4 files. one at a time.
@cvvergara Ok,After I split the file, it succeeded. Thanks!
@hetongde great!
Hi @cvvergara ,
This is the
bad_alloc
issue with osm_relations we was reviewing yesterday in your branch https://github.com/cvvergara/osm2pgrouting/tree/test2With this commit https://github.com/cvvergara/osm2pgrouting/commit/32c7a7bf4f2c309366c255b9eac33f696aba02d9 the bad_alloc does not raise because you don't take all the relations.
This happened in my dockerized environment (on a Ubuntu host). In the next days I'll try to reproduce this error with the same dataset but without Docker. Dataset: https://drive.google.com/open?id=1IyRLf0mjzFI5fx9y2WO5DboFjTw8X--k
Command line:
Shell output:
Thanks,