A piece of Question about ADIOS converter bp2ncd

[Lucymugua]

I am using ADIOS in my research work. However, I got "Error:NetCDF: Name contains illegal characters" when I using ADIOS converter bp2ncd to convert bp file format to NetCDF format.

The version of NetCDF is netcdf/4.4-icc16-mvapich2.

Could anyone give me some advance on this error?

[pnorbert]

Hi,

The error indicates that you are using some characters in your variable names that the netcdf format does not accept. An example is '/' since paths are not allowed at all.

BTW, the bp2ncd tool is ancient and is not supported. No one has used it for anything meaningful for so long, we just dropped to care about it. Why do you need that conversion for your research?

Best regards

On Tue, Mar 20, 2018 at 9:42 AM, Lucymugua notifications@github.com wrote:

I am using ADIOS in my research work. However, I got "Error:NetCDF: Name contains illegal characters" when I using ADIOS converter bp2ncd to convert bp file format to NetCDF format.

The version of NetCDF is netcdf/4.4-icc16-mvapich2.

Could anyone give me some advance on this error?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ornladios/ADIOS/issues/172, or mute the thread https://github.com/notifications/unsubscribe-auth/ADGMLXfvAzGmOix3EEKJSswHd-FR5dxFks5tgQdJgaJpZM4Sx6le .

[Lucymugua]

hi pnorbert,

Thanks for you kindly reply.

I have checked all of my variable names in the xml file and found that they contain only letters, numbers and underscores("_") which are allowed in using this tool. Futhermore, I am using ADIOS 1.12.0 which is the latest version. And there is also a part about bp2ncd tool in ADIOS 1.12.0 user guide which is shown in the above picture. If it have been dropped, why there is also an introduction about it in the latest version user manual? And what do we commonly use to convert bp file format to NetCDF or binary file format currently?

Thanks, Lucy

[pnorbert]

Lucy,

Can I see your xml and the "bpls -lav" output for your file? Make sure the "path" attribute in each element is "" or does not exist.

You are right with your criticism, we either should completely remove this tool from the release or stop staying it's not supported. I meant it may or may not work correctly as it has not been used for years and is not tested regularly.

Best regards Norbert

On Tue, Mar 20, 2018 at 9:31 PM, Lucymugua notifications@github.com wrote:

hi pnorbert,

Thanks for you kindly reply.

I have checked all of my variable names in the xml file and found that they contain only letters, numbers and underscores("_") which are allowed in using this tool. [image: image] https://user-images.githubusercontent.com/37583049/37690973-7fdfd2a0-2ce9-11e8-9e8c-b7c792ef7651.png Futhermore, I am using ADIOS 1.12.0 which is the latest version. And there is also a part about bp2ncd tool in ADIOS 1.12.0 user guide which is shown in the above picture. If it have been dropped, why there is also an introduction about it in the latest version user manual? And what do we commonly use to convert bp file format to NetCDF or binary file format currently?

Thanks, Lucy

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ornladios/ADIOS/issues/172#issuecomment-374811479, or mute the thread https://github.com/notifications/unsubscribe-auth/ADGMLREDJluPfzfsZVoAvQM5evE8cd7Wks5tga1ZgaJpZM4Sx6le .

[Lucymugua]

Dear Norbert,

Thanks for your kind reply. My xml file, output file structure and bpls -lav outfile results are listed below:

1.xml file:

2.output file strcture:

3.bpls -lav results:

Thanks for your help.

Best regards, Lucy

[pnorbert]

Lucy,

Your troubles are probably caused by the adios timers in the file. Please rebuild adios for your runs turning off the timers. Use the options "--disable-timers --disable-timer-events" when configuring the build with autoconf tools, or "export adios_timers=OFF" when using cmake.

If you need this existing dataset, there may be a way to read-in/write-out again using examples/staging/stage_write, after building that with a timer-less adios installation.

Best regards Norbert

On Thu, Mar 22, 2018 at 11:29 PM, Lucymugua notifications@github.com wrote:

Dear Norbert,

Thanks for your kind reply. My xml file, output file structure and bpls -lav outfile results are listed below:

1.xml file: [image: image] https://user-images.githubusercontent.com/37583049/37810048-37aa0906-2e8d-11e8-8777-38dc781c1248.png

[image: image] https://user-images.githubusercontent.com/37583049/37810072-4790f5f0-2e8d-11e8-80e4-bc6aa99706cd.png

2.output file strcture: [image: image] https://user-images.githubusercontent.com/37583049/37810001-e6a74e06-2e8c-11e8-8cfa-be5d9281b61d.png

[image: image] https://user-images.githubusercontent.com/37583049/37810014-00fdb358-2e8d-11e8-9e48-c89c645d28ca.png

3.bpls -lav results: [image: image] https://user-images.githubusercontent.com/37583049/37809955-a38d50b6-2e8c-11e8-8bd2-15af5799d4d4.png

[image: image] https://user-images.githubusercontent.com/37583049/37809968-b1bfeec8-2e8c-11e8-99c1-d33ee800e47c.png

Thanks for your help.

Best regards, Lucy

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ornladios/ADIOS/issues/172#issuecomment-375533357, or mute the thread https://github.com/notifications/unsubscribe-auth/ADGMLS1KPxOpdcGTTUo2pzfRzuEVYzGQks5thGwhgaJpZM4Sx6le .

[pnorbert]

Lucy, My suggestion will still not help. bp2ncd is so old and unmaintained that it does not work with the subfiles generated by the aggregation method. I am looking for some solution but at the moment I have nothing to offer. Sorry Norbert

On Fri, Mar 23, 2018 at 9:22 AM, Norbert Podhorszki < norbert.podhorszki@gmail.com> wrote:

Lucy,

Your troubles are probably caused by the adios timers in the file. Please rebuild adios for your runs turning off the timers. Use the options "--disable-timers --disable-timer-events" when configuring the build with autoconf tools, or "export adios_timers=OFF" when using cmake.

If you need this existing dataset, there may be a way to read-in/write-out again using examples/staging/stage_write, after building that with a timer-less adios installation.

Best regards Norbert

On Thu, Mar 22, 2018 at 11:29 PM, Lucymugua notifications@github.com wrote:

Dear Norbert,

Thanks for your kind reply. My xml file, output file structure and bpls -lav outfile results are listed below:

1.xml file: [image: image] https://user-images.githubusercontent.com/37583049/37810048-37aa0906-2e8d-11e8-8777-38dc781c1248.png

[image: image] https://user-images.githubusercontent.com/37583049/37810072-4790f5f0-2e8d-11e8-80e4-bc6aa99706cd.png

2.output file strcture: [image: image] https://user-images.githubusercontent.com/37583049/37810001-e6a74e06-2e8c-11e8-8cfa-be5d9281b61d.png

[image: image] https://user-images.githubusercontent.com/37583049/37810014-00fdb358-2e8d-11e8-9e48-c89c645d28ca.png

3.bpls -lav results: [image: image] https://user-images.githubusercontent.com/37583049/37809955-a38d50b6-2e8c-11e8-8bd2-15af5799d4d4.png

[image: image] https://user-images.githubusercontent.com/37583049/37809968-b1bfeec8-2e8c-11e8-99c1-d33ee800e47c.png

Thanks for your help.

Best regards, Lucy

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ornladios/ADIOS/issues/172#issuecomment-375533357, or mute the thread https://github.com/notifications/unsubscribe-auth/ADGMLS1KPxOpdcGTTUo2pzfRzuEVYzGQks5thGwhgaJpZM4Sx6le .

[Lucymugua]

Dear Norbert,

Thanks for your solution. I am planning to write a program about convert bp format to NetCDF format on my own without using the tool.

BTW, is there any other kind of tool have the same function as bp2ncd to do the conversion?

Best regards, Lucy

[pnorbert]

Hi Lucy,

If you are comfortable with python/numpy and building libraries for python, then probably that's the easiest way to do the conversion. ADIOS has a numpy wrapper (wrapper/numpy/ in the source that has to built and installed for python separately). There is also a netcdf4 package for python/numpy http://unidata.github.io/netcdf4-python/ https://pypi.python.org/pypi/netCDF4

If you want to work in C, then there is the examples/staging/stage_write/stage_write.c code that can read any BP file of global arrays in parallel (you define the decomposition for reading) and then writes out again using any ADIOS methods. Replacing the last part for NetCDF should work for you. Also, your variables are fairly small, so this should run fine serial.

Best regards Norbert

On Sat, Mar 24, 2018 at 9:13 AM, Lucymugua notifications@github.com wrote:

Dear Norbert,

Thanks for your solution. I am planning to write a program about convert bp format to NetCDF format on my own without using the tool.

BTW, is there any other kind of tool have the same function as bp2ncd to do the conversion?

Best regards, Lucy

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ornladios/ADIOS/issues/172#issuecomment-375888430, or mute the thread https://github.com/notifications/unsubscribe-auth/ADGMLXEV8CK2ccEptxE4cLSY2IhCWZqgks5thkZmgaJpZM4Sx6le .

[jychoi-hpc]

I have been working on developing the Adios python wrapper and made an example to convert ncdf to bp file (wrappers/numpy/example/utils/ncdf2bp.py). If you need, I can help to write "bp2ncdf.py". If this is the case, it will be great to have an example ncdf file.

Best regards, Jong

-- Jong Youl Choi Scientific Data Group Computer Science and Math Division Oak Ridge National Laboratory Homepage: http://www.ornl.gov/~jyc/

On Mar 24, 2018, at 9:47 AM, pnorbert notifications@github.com wrote:

Hi Lucy,

If you are comfortable with python/numpy and building libraries for python, then probably that's the easiest way to do the conversion. ADIOS has a numpy wrapper (wrapper/numpy/ in the source that has to built and installed for python separately). There is also a netcdf4 package for python/numpy http://unidata.github.io/netcdf4-python/ https://pypi.python.org/pypi/netCDF4

If you want to work in C, then there is the examples/staging/stage_write/stage_write.c code that can read any BP file of global arrays in parallel (you define the decomposition for reading) and then writes out again using any ADIOS methods. Replacing the last part for NetCDF should work for you. Also, your variables are fairly small, so this should run fine serial.

Best regards Norbert

On Sat, Mar 24, 2018 at 9:13 AM, Lucymugua notifications@github.com wrote:

Dear Norbert,

Thanks for your solution. I am planning to write a program about convert bp format to NetCDF format on my own without using the tool.

BTW, is there any other kind of tool have the same function as bp2ncd to do the conversion?

Best regards, Lucy

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ornladios/ADIOS/issues/172#issuecomment-375888430, or mute the thread https://github.com/notifications/unsubscribe-auth/ADGMLXEV8CK2ccEptxE4cLSY2IhCWZqgks5thkZmgaJpZM4Sx6le .

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[Lucymugua]

Hi Norbert,

Thank you so much for your advice. I guess what you referred are following: /adios-1.12.0/wrappers/numpy/example/utils/ncdf2bp.py Jong made the example to convert ncdf to bp file And he said he can help to write "bp2ncdf.py".

Best regards, Lucy

[Lucymugua]

Hi Jong,

I would greatly appreciate it if you can help to write"bp2ncdf.py". I really need that kind of tool to help do the format conversion.

Best regards, Lucy

[jychoi-hpc]

Hi, Lucy,

I have just added "bp2ncdf.py" in wrappers/numpy/example/utils. The usage is as follows: usage: bp2ncdf.py [-h] [-v] BPFILE OUTFILE

This is the first version and may have some bugs. Please feel free to report any problem.

Best regards, Jong

-- Jong Youl Choi Scientific Data Group Computer Science and Math Division Oak Ridge National Laboratory Homepage: http://www.ornl.gov/~jyc/

On Mar 27, 2018, at 4:32 AM, Lucymugua notifications@github.com wrote:

Hi Jong,

I would greatly appreciate it if you can help to write"bp2ncdf.py". I really need that kind of tool to help do the format conversion.

Best regards, Lucy

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[Lucymugua]

Hi Jong,

   I am sorry for not replying you for such a long time. During these days, I am trying to write a data transform program designed for my own program. However, when I use adios_selection_boundingbox method, it has some error:

I don 't know what is wrong. I am wondering is this method still be using?

Best regards, Lucy